Constraint relationship for use in an image segregation

ABSTRACT

A soft, weighted constraint imposed upon image locations can be used to provide a more accurate segregation of an image into intrinsic material reflectance and illumination components. The constraint is arranged to constrain all color band variations between the image locations into one integral constraining relationship.

This is a Continuation application of U.S. patent application Ser. No.14/273,184, filed May 8, 2014 and hereby incorporated by referenceherein.

BACKGROUND OF THE INVENTION

Many significant and commercially important uses of modern computertechnology relate to images. These include image processing, imageanalysis and computer vision applications. In computer visionapplications, such as, for example, object recognition and opticalcharacter recognition, it has been found that a separation ofillumination and material aspects of an image can significantly improvethe accuracy and effectiveness of computer performance. Significantpioneer inventions related to the illumination and material aspects ofan image are disclosed in U.S. Pat. No. 7,873,219 to Richard MarkFriedhoff, entitled Differentiation Of Illumination And ReflectionBoundaries and U.S. Pat. No. 7,672,530 to Richard Mark Friedhoff et al.,entitled Method And System For Identifying Illumination Flux In An Image(hereinafter the Friedhoff patents).

SUMMARY OF THE INVENTION

The present invention provides a method and system comprising imageprocessing techniques that utilize spatio-spectral information relevantto an image, derived from multiple sets of selectively variedrepresentations of the image to accurately and correctly identifyillumination and material aspects of the image. In an exemplaryembodiment of the present invention, a soft, weighted constraint imposedupon image locations can be used to provide a more accurate segregationof an image into intrinsic material reflectance and illuminationcomponents. The constraint is arranged to constrain all color bandvariations between the image locations into one integral constrainingrelationship.

In a first exemplary embodiment of the present invention, an automated,computerized method is provided for processing an image. According to afeature of the present invention, the method comprises the steps ofproviding an image file depicting the image, including image locations,in a computer memory, selecting image locations i and j, setting aconstraint between the selected image locations i and j according to arelationship defined by a bi-illuminant, dichromatic reflection model(BIDR model), the constraint set as a function of a degree of changecorresponding to each of an illumination change and/or a materialreflectance change according to a sigmoid function based upon the BIDRmodel, identifying an intrinsic component of the selected imagelocations as a function of the constraint and generating an intrinsicimage corresponding to the image as a function of the intrinsiccomponent.

In a second exemplary embodiment of the present invention, a device isprovided. The device comprises a computer and a memory storing an imagefile containing an image having image locations. According to a featureof the present invention, the computer is arranged and configured toexecute a routine to select image locations i and j, set a constraintbetween the selected image locations i and j according to a relationshipdefined by a bi-illuminant, dichromatic reflection model (BIDR model),the constraint set as a function of a degree of change corresponding toeach of an illumination change and/or a material reflectance changeaccording to a sigmoid function based upon the BIDR model, identify anintrinsic component of the selected image locations as a function of theconstraint, and generate an intrinsic image corresponding to the imageas a function of the intrinsic component.

In a third exemplary embodiment of the present invention, a computerprogram product is provided. The computer program product is disposed ona non-transitory computer readable media, the product including computerexecutable process steps operable to control a computer to: receive animage file depicting an image having image locations, select imagelocations i and j, set a constraint between the selected image locationsi and j according to a relationship defined by a bi-illuminant,dichromatic reflection model (BIDR model), the constraint set as afunction of a degree of change corresponding to each of an illuminationchange and/or a material reflectance change according to a sigmoidfunction based upon the BIDR model, identify an intrinsic component ofthe selected image locations as a function of the constraint, andgenerate an intrinsic image corresponding to the image as a function ofthe intrinsic component.

In accordance with yet further embodiments of the present invention,computer systems are provided, which include one or more computersconfigured (e.g., programmed) to perform the methods described above. Inaccordance with other embodiments of the present invention, computerreadable media are provided which have stored thereon computerexecutable process steps operable to control a computer(s) to implementthe embodiments described above. The automated, computerized methods canbe performed by a digital computer, analog computer, optical sensor,state machine, sequencer or any device or apparatus that can be designedor programmed to carry out the steps of the methods of the presentinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computer system arranged and configuredto perform operations related to images.

FIG. 2 shows an n×m pixel array image file for an image stored in thecomputer system of FIG. 1.

FIG. 3a is a data flow block diagram for an image segregation operationperformed according to an exemplary embodiment of the present inventionincluding a set of selectively varied representations of the image ofthe image file of FIG. 2.

FIG. 3b shows a graphic representation of a scale-spaced pyramid as theset of selectively varied representations of FIG. 3 a.

FIG. 3c is a data flow block diagram for an image segregation operationaccording to a further exemplary embodiment of the present inventionincluding multiple sets of selectively varied representations of theimage of the image file of FIG. 2.

FIG. 3d is a flow chart for identifying blend pixels in an image.

FIG. 3e is a flow chart showing a shadow boundary detection step, fromthe flow chart of FIG. 3 d.

FIG. 3f is a flow chart for a texture detection step, from the flowchart of FIG. 3 d.

FIG. 3g is a flow chart for a color flow vector step of FIG. 3 c.

FIG. 4 is a functional block diagram for an image segregation systemarchitecture for performing the data flow of FIG. 3a , implemented inthe computer system of FIG. 1, according to a feature of the presentinvention.

FIG. 5 shows a graphical user interface for use in connection with animplementation of the image segregation system architecture feature ofthe present invention.

FIG. 6a is a flow chart for identifying Type C token regions in theimage file of FIG. 2a , according to a feature of the present invention.

FIG. 6b is an original image used as an example in the identification ofType C tokens.

FIG. 6c shows Type C token regions in the image of FIG. 6 b.

FIG. 6d shows Type B tokens, generated from the Type C tokens of FIG. 6c, according to a feature of the present invention.

FIG. 7 is a flow chart for a routine to test Type C tokens identified bythe routine of the flow chart of FIG. 6a , according to a feature of thepresent invention.

FIG. 8 is a flow chart for constructing Type B tokens via an arbitraryboundary removal technique, according to a feature of the presentinvention.

FIG. 9 is a flow chart for creating a token graph, containing token mapinformation, according to a feature of the present invention.

FIG. 10 is a flow chart for constructing Type B tokens via an adjacentplanar token merging technique, according to a feature of the presentinvention.

FIG. 11 is a flow chart for generating Type C tokens via a local tokenanalysis technique, according to a feature of the present invention.

FIG. 12 is a flow chart for constructing Type B tokens from Type Ctokens generated via the local token analysis technique of FIG. 11,according to a feature of the present invention.

FIG. 13 is a graphic representation of a log color space chromaticityplane according to a feature of the present invention.

FIG. 14 is a flow chart for determining a list of colors depicted in aninput image.

FIG. 15 is a flow chart for determining an orientation for a logchromaticity space, according to a feature of the present invention.

FIG. 16 is a flow chart for determining log chromaticity coordinates forthe colors of an input image, as determined through execution of theroutine of FIG. 14, according to a feature of the present invention.

FIG. 17 is a flow chart for augmenting the log chromaticity coordinates,as determined through execution of the routine of FIG. 16, according toa feature of the present invention.

FIG. 18 is a flow chart for clustering the log chromaticity coordinates,according to a feature of the present invention.

FIG. 19 is a flow chart for assigning the log chromaticity coordinatesto clusters determined through execution of the routine of FIG. 18,according to a feature of the present invention.

FIG. 20 is a flow chart for detecting regions of uniform reflectancebased on the log chromaticity clustering according to a feature of thepresent invention.

FIG. 21 is a representation of an [A] [x]=[b] matrix relationshipaccording to an exemplary embodiment of the present invention.

FIG. 22 is a representation of an [A] [x]=[b] matrix relationshipaccording to a further exemplary embodiment of the present invention.

FIG. 23 is a flow chart for a weighted constraint according to a featureof the present invention.

FIG. 24 shows a graph for a sigmoid function indication of a weight,according to a feature of the present invention.

FIG. 25a shows spaced locations in one level of the pyramid of FIG. 3 b.

FIG. 25b shows spaced locations between temporally spaced frames of avideo.

FIGS. 26a and b show graphs for scaled threshold values in each of anglemetric and distance metric modes, respectively.

FIG. 27 is a flow chart for a simple mean method for creating ascale-spaced pyramid according to a feature of the present invention.

FIG. 28 is a graphic representation of pixel locations at differentscales of a scale-spaced pyramid, as shown in FIG. 3 b.

FIG. 29 is a flow chart for a temporal constraint.

FIG. 30 is a flow chart for an anchor constraint.

FIG. 31 is a functional block diagram for a service provider componentfor use in the image segregation system architecture of FIG. 4.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to the drawings, and initially to FIG. 1, there is shown ablock diagram of a computer system 10 arranged and configured to performoperations related to images. A CPU 12 is coupled to a device such as,for example, a digital camera 14 via, for example, a USB port. Thedigital camera 14 operates to download images stored locally on thecamera 14, to the CPU 12. The CPU 12 stores the downloaded images in amemory 16 as image files 18. The image files 18 can be accessed by theCPU 12 for display on a monitor 20, or for print out on a printer 22.

Alternatively, the CPU 12 can be implemented as a microprocessorembedded in a device such as, for example, the digital camera 14 or arobot. The CPU 12 can also be equipped with a real time operating systemfor real time operations related to images, in connection with, forexample, a robotic operation or an interactive operation with a user.

As shown in FIG. 2, each image file 18 comprises an n×m pixel array.Each pixel, p, is a picture element corresponding to a discrete portionof the overall image. All of the pixels together define the imagerepresented by the image file 18. Each pixel comprises a digital valuecorresponding to a set of color bands, for example, red, green and bluecolor components (RGB) of the picture element. The present invention isapplicable to any multi-band image, where each band corresponds to apiece of the electro-magnetic spectrum. The pixel array includes n rowsof m columns each, starting with the pixel p(1,1) and ending with thepixel p(n, m). When displaying or printing an image, the CPU 12retrieves the corresponding image file 18 from the memory 16, andoperates the monitor 20 or printer 22, as the case may be, as a functionof the digital values of the pixels in the image file 18, as isgenerally known.

According to a feature of the present invention, in an image process,the CPU 12 operates to analyze and process information, for example, theRGB values of the pixels of an image stored in an image file 18, toachieve various objectives, such as, for example, a segregation ofillumination and material aspects of a scene depicted in the image of animage file 18, to generate corresponding intrinsic images. The intrinsicimages include, for example, an illumination image, to capture theintensity and color of light incident upon each point on the surfacesdepicted in the image, and a material reflectance image, to capturereflectance properties of surfaces depicted in the image (the percentageof each wavelength of light a surface reflects). The separation ofillumination from material in the intrinsic images provides the CPU 12with images optimized for more effective and accurate further processingin such applications as computer vision.

To that end, FIG. 3a shows a data flow diagram for an image segregationoperation performed according to an exemplary embodiment of the presentinvention. In block 1000, an image file 18 is selected and input to theCPU 12. In block 1002, the CPU 12 operates to form a set of selectivelyvaried representations of the image, depicted in image files 18 a, 18 b,. . . 18 n, from the selected image file 18. The variations among therepresentations can be based upon any preselected spacial, spectraland/or temporal groupings of the original image. The variations areselected so as to provide image representations most suitable forefficient and accurate analysis of illumination and material aspects ofthe image.

In one preferred embodiment of the present invention, computer operationfor illumination and material image segregation can be performed at arelatively coarse scale representation of the image, and then used toconstrain a solution at a relatively finer scale resolution. Byenforcing a segregation that is consistent across different scales ofresolution, performance of a segregation operation is not only morespeedy and efficient, but also more accurate.

In another preferred embodiment, a segregation of illumination andmaterial aspects of the image is selectively divided amongrepresentations at differing scales of image resolution. Differentfeatures of a scene depicted in an image can have different intrinsicscales. For example, edges in an image have an intrinsic scale mosteffectively analyzed at a fine scale of resolution, while non-edgedregions, such as regions of uniform material, can be accurately analyzedat relatively coarse scales of resolution. Thus, an image is divided byedge and non-edge regions, segregating the edge regions at a fine scaleof resolution, and the remaining non-edge regions at a relatively coarsescale of resolution. More generally, pixels of the image (or otherpreselected regions of the image) can each be divided according to theintrinsic scale of the respective pixel, and processed at arepresentation of the image at an appropriate scale. The division ofsegregation operations in this manner provides a sequence of simplesolution systems, rather than one relatively large complex system tosolve.

Accordingly, for example, in one exemplary embodiment of the presentinvention, the CPU 12 can form multi-resolution representations such asa scale-spaced pyramid of representations of the image. As shown in FIG.3b , in a scale-spaced pyramid, the set of image files 18 a, b, c . . .n, are generated, each at a different scale of resolution relative tothe resolution of the original image, from a finest resolution (definedas the resolution of the original image in image file 18) to relativelycoarser resolutions, at each upper level of the pyramid. Each of imagefiles 18, 18 a, . . . 18 n of FIG. 3b is an array of pixels, asgenerally shown in FIG. 2. Image file 18 a is a coarser resolutionversion of the original image file 18, and image file 18 b is a coarserresolution version of image file 18 a, and so on. For example, asillustrated in FIG. 3b , pixels P(5, 3), P(6, 3), P(5, 4) and P(6, 4) offine resolution image file 18 map to, and therefor correspond to, asingle pixel, P_(a)(3, 2) in the relatively coarse resolution image file18 a. A downsampling scale D indicates the difference in scale betweenlevels of the pyramid such that a higher level, low resolutionrepresentation of the image has 1/D² fewer pixels than the lower, highresolution level.

A scale-spaced pyramid(s) is formed via well known Gaussian and/orLaplacian pyramid formation techniques. In a Gaussian pyramid, theoriginal image file 18 is convolved with a Gaussian kernel g, for a lowpass filtered, subsampled version of the original image (image file 18a), having a cut off frequency set by the parameter σ. A full Gaussianpyramid is formed when each coarse scale (e.g. image file 18 a) isconvolved with the Gaussian kernal to form a next level (image file 18b), and so on (to image file 18 n). A Laplacian pyramid provides aseries of bandpass filtered versions of the original image, wherein eachscale of the pyramid corresponds to a different band of imagefrequencies.

If a series of selectively varied images represented by image files 18,18 a, . . . 18 n is stated as a Gaussian pyramid of low passed filteredversions of the original image, and expressed by original image i₀, andscales i₁, i₂, . . . i_(n), respectively, then any level of the pyramidis related to a downsampled next lower (finer) level by:i_(n)=⇓(g*i_(n−1)), where ⇓ denotes a downsampling, g represents a lowpass filter, such as, for example, the Gaussian low pass filter and *denotes a spatial convolution.

For a Laplacian pyramid, in the series of images represented by imagefiles 18, 18 a, . . . 18 n, each level of the pyramid i_(n) can beexpressed in terms of a difference between levels of the Gaussianpyramid: i_(n)=i_(n)−⇑i_(n+1), where ⇑ denotes upsampling.

Moreover, starting with a multiplicative model for the image: I=M×L,where I is the original image, M is the material component of the imageand L is the illumination component, it can be shown that each level ofthe pyramid can be decomposed into a material pyramid and anillumination pyramid. For an additive model, the log color values arelog I=log M+log L. In terms of the Gaussian pyramid, the decompositioninto material and illumination components can be expressed as: g_(σ)*logI=g_(σ)*log M+g_(σ)*log L.

In block 1004, the CPU 12 operates to perform an image segregationoperation on at least one of the representations of the pyramid, forexample, original image file 18 and image file 18 a, to generateintrinsic images including, for example, each of an illumination imageand a material reflectance image for each selected scale of the image.Each of the intrinsic images corresponds to the image depicted at theresolution scales of image file 18 and image file 18 a, respectively.According to a feature of the present invention, the image segregationoperation performed by the CPU 12 at each scale of resolution, isimplemented according to a spatio-spectral operator/constraint/solvermodel to identify the illumination and material reflectance componentsof the original image or any representation of the original image.

FIG. 3c illustrates a further exemplary embodiment of the presentinvention. As shown, the concept of selectively varying the image file18, as illustrated in FIGS. 3a & b, is implemented in a manner toorganize image representations based upon multiple, differentpreselected image characteristics, for example, spacial, spectral andtemporal groupings of the original image. In an exemplary embodiment ofthe present invention, an input image 18, as shown in FIG. 2, ispre-processed (block 1008). The pre-processing can include such optionaloperations as the correction of chromatic aberration according to thetechniques taught in U.S. Pat. No. 7,865,031, and a step to linearizeimage data, for more accurate processing.

Thereafter, an image pyramid, as shown in FIG. 3b is created, forexample, using either a Gaussian or Laplacian pyramid formationtechnique, as described above (blocks 1010, 1012). According to afurther exemplary embodiment of the present invention, blocks 1010 and1012 are implemented using a simple mean method. Each of the knownGaussian or Laplacian pyramid formation techniques for creating thepyramids, as described above, tend to average or blur colors from a highresolution to the next, lower resolution level. In some instances, animage analysis, according to a feature of the present invention, reliesupon purity of color values for an accurate measure of color change. Inthose instances, a pyramid arrangement that preserves the purity ofcolor from scale to scale is needed to insure the accuracy of solveresults. Thus, according to a feature of the present invention, toincrease accuracy in a solve based upon color change, a simple meanmethod, as described below with reference to FIG. 27, is used to assurecolor purity between the pyramid levels. Each scale representation ofthe image 18, 18 a, . . . 18 n, is then processed to divide each imagerepresentation into Type C tokens, as will be described in detail, belowwith reference to FIG. 6a (blocks 1014, 1016).

As will be discussed in detail, a Type C token comprises a connectedimage region of similar image properties among the contiguous pixels ofthe token, for example, similar color and intensity, where similarity isdefined with respect to a noise model for the imaging system used torecord the image. The use of Type C tokens improves processingefficiency.

According to a feature of the present invention, after thepre-processing of block 1008, the CPU 12 is operated to create, as asecond grouping of pixels, a blend pixel image from the image depictedin the input image file 18 (block 1018). Blend pixels are generallypositioned at material boundaries, and depict a blend of the differentcolors of the adjacent different materials forming the boundary. Assuch, each blend pixel depicts a color that does not actually exist inthe scene depicted in the image file 18, and thus, can degrade theaccuracy of image processing to segregate the image file 18 intocorresponding intrinsic illumination and material reflectance images.

Referring now to FIG. 3d , there is shown a flow chart for identifyingblend pixels in an image. An original image file 18 is processed toidentify Type C tokens, for example, according to the process stepsshown in FIG. 6a , as will be described below (block 1100). In step1102, the CPU 12 is operated to examine each Type C token, to identifyall one dimensional (1-D) Type C tokens (that is a Type C tokenconsisting of a single string of pixels). This is because tokensconsisting of blend pixels are created by a mixing of different materialcolors at a material boundary, and tend to be one dimensional.

If a current Type C token being examined by the CPU 12 is not a 1-Dtoken, the token is not labeled as a blend pixel token (block 1104). Ifthe current Type C token is a 1-D token, the token is added to a list of1-D tokens, and the CPU 12 proceeds to step 1106. In step 1106, the CPU12 operates to examine the current 1-D Type C token to determine if thetoken is on a shadow boundary caused by an illumination change, or stilla candidate for a material based blend pixel token. FIG. 3e shows a flowchart for each of a distance-based method and an angle-based method, forexecuting a shadow boundary analysis on a candidate 1-D token.

In step 1200, an indication is set, for example, by a user, to indicateexecution by the CPU 12 of either the distance-based or angle-basedmethods. If step 1200 is set to the distance-based method, the CPU 12proceeds to the distance-based method 1202. In the distance-based method1202, the CPU 12, in series or parallel, computes each of a logchromaticity image (1204) and a mean similarity weights s_(i) (1206).

A method and system for separating illumination and reflectance using alog chromaticity image is disclosed in U.S. Pat. No. 7,596,266, which ishereby expressly incorporated by reference. In step 1204, the CPU 12executes the methods taught in U.S. Pat. No. 7,596,266, to compute anillumination invariant log chromaticity image based upon the imagedepicted in the image file 18, being processed. The processing can alsoinclude the determination of a BIDR normal (block 1024), as required forthe computation of the log chromaticity image, according to the methodsdescribed in U.S. Pat. No. 7,596,266.

If the candidate 1-D token is caused by an illumination change, acorresponding gradient for each pixel in the candidate 1-D token willhave a large value in the original image of the image file 18. However,the gradients for the pixels of the candidate 1-D token will have arelatively small value in the corresponding illumination invariant, logchromaticity image.

According to the distance-based method feature of the present invention,the gradient relationship, as between the original image and thecorresponding, illumination invariant log chromaticity image, is used todetect whether the candidate 1-D token is caused by a shadow. To thatend, in step 1206, the CPU 12 computes weighted variance values (V) fortokens in a small neighborhood on either side of a candidate 1-D token,as follows: V=Σ_(i,j=1) ^(N)s_(ij)(I_(ij)−M)²/Σ_(i,j=1) ^(N)s_(ij),where N is the selected size of the neighborhood used to perform theanalysis, M is the mean variance for a token i,j relative to the smallneighborhood defining a potential fully lit and fully shadowed regionsaround the candidate 1-D token, is the recorded color and intensity forthe respective token of the local neighborhood, and s_(ij) is asimilarity weight, expressed as follows: s_(ij)∝1/max (minVal,(I_(ij)−M)²) with minVal being set at a value to avoid division by zeroor a value close to zero.

In steps 1208 and 1210, respectively, the CPU 12 computes weightedvariances in each of the log chromaticity image, V_(BIDR) and theoriginal image, V_(ORIG) of the image file 18 for the selected smallneighborhood surrounding the candidate 1-D token.

In step 1212, the CPU 12 compares V_(ORIG) to a first threshold value,thres1 and compares V_(BIDR) to a second threshold value, thres2. IfV_(ORIG) is greater than the thres1, a threshold set to indicate arelatively large change in gradient values for the pixels of thecandidate 1-D token, in the original image, and V_(BIDR) is less thanthres2, a threshold set to indicate a relatively small change ingradient values for the pixels of the candidate 1-D token in theillumination invariant log chromaticity image, then the CPU 12 finds anindication for a token at a boundary caused by an illumination change,and returns an indication of not a blend pixel token (step 1214). If theopposite is true, the CPU 12 returns an indication of a blend pixeltoken candidate (step 1216).

If step 1200 is set to the angle-based method, the CPU 12 proceeds tothe angle-based method 1220. In step 1222, the CPU 12 operates togenerate N images, each based upon the image depicted in the image file18, and each of the N images being generated with a different gaussianblur sigma, sigma 1 to sigma N, respectively. In steps 1224 a to 1224 n,the CPU 12 computes an angle between a gradient for the candidate 1-Dtoken, in each one of the blurred versions of the image, and the BIDRnormal (1024).

As fully described in U.S. Pat. No. 7,596,266, a bi-illuminant,dichromatic reflection (BIDR) model predicts that differing colormeasurement values fall within a cylinder in RGB space, from a dark end(in shadow) to a bright end (lit end), along a positive slope, when thecolor change is due to an illumination change forming a shadow over asingle material of a scene depicted in the image. The BIDR normalcorresponds to the orientation in, for example, the RGB color space, ofthe positive slope for the color variation caused by the illuminationchange across the single material. Accordingly, if the candidate 1-Dtoken has a gradient that has an orientation in the color space that isclose to the orientation of the BIDR normal (small angle), it is likelyformed at a shadow boundary, rather than at a material boundary.

In step 1226, the CPU 12 operates to identify the minimum angle found insteps 1224 a to 1224 n, Ang_(min). In step 1228, the CPU 12 comparesAng_(min) to a threshold value. The threshold is set at a value toreflect an angle that shows substantial alignment to the BIDR normal,and thus, color change due to illumination. Thus, if Ang_(min) is lessthan the threshold, the CPU 12 returns an indication of not a blendpixel token (step 1214). If the opposite is true, the CPU 12 returns anindication of a blend pixel token candidate (step 1216).

In alternative embodiments of the present invention, instead of aminimum angle in step 1226, the CPU 12 identifies a differentstatistical metric, such as, for example, a median or variance. Also,instead of a hard, fixed threshold, a soft threshold function isimplemented.

When the CPU 12 enters return step 1214, the CPU then proceeds to step1104 (FIG. 3d ), and removes the candidate 1-D token from the list of1-D tokens. When the CPU 12 enters a return step 1216, the CPU 12proceeds to step 1108 (FIG. 3d ). In step 1108, the CPU 12 operates toexamine the current 1-D Type C token to determine if the token is atexture token, or is in fact, a material based blend pixel token. FIG.3f is a flow chart for a texture detection step, (step 1108 from theflow chart of FIG. 3d ).

In step 1300, the CPU 12 receives the candidate 1-D token, from theshadow analysis routine (1106). In step 1302, the CPU 12 operates tocalculate the centroid of the candidate 1-D token. In steps 1304 and1306, the CPU 12 operates to calculate the average color for the pixelsof the candidate 1-D token, and compares that color to pixels of apre-selected neighborhood surrounding the candidate 1-D token, todetermine the number of pixels in the neighborhood, Ns, that match thecolor of the candidate 1-D token.

A texture in an image consists of a repeating pattern of colors. If thecandidate 1-D token is within a texture, rather than at a materialboundary, Ns will have a value greater than a threshold value set at alevel that reflects a repeating pattern on either side of the candidatetoken, as opposed to two different materials at a material boundary. IfNs is greater than the threshold, thres, the CPU 12 returns anindication of a texture token (step 1310). If Ns is lower than thethreshold, the CPU 12 returns an indication of a blend pixel token (step1312).

When the CPU 12 enters return step 1310, the CPU then proceeds to step1104 (FIG. 3d ), and removes the candidate 1-D token from the list of1-D tokens. When the CPU 12 enters a return step 1312, the CPU 12proceeds to step 1110 (FIG. 3d ).

In step 1110, the CPU 12 marks the candidate 1-D token as a blend pixeltoken in the 1-D token list. After executing steps 1100-1110 of theroutine of FIG. 3d for each token of the image file 18, being processed,the CPU 12 proceeds to step 1112. The CPU applies the list of blendtokens identified through execution of the routine of FIG. 3d , tocreate a blend pixel mask (1114). The blend pixel mask completes block1018 (FIG. 3c ) and provides a representation of the image depicted inthe image file 18, the representation set by the CPU 12 to depict therecorded intensity values, for example, RGB values, for each pixellocation in the image identified as a blend pixel, with a nil value ateach other pixel location.

Referring once again to FIG. 3c , in block 1020, the CPU 12 operates tocreate a scale-spaced pyramid of the blend pixel representation of theimage, executing the same routines described above used to create thescale-spaced pyramid for the original image (step 1012).

As noted above, a BIDR Model predicts that differing color measurementvalues fall within a cylinder in RGB space, from a dark end (in shadow)to a bright end (lit end), along a positive slope, when the color changeis due to an illumination change forming a shadow over a single materialof a scene depicted in the image. The BIDR normal corresponds to theorientation in, for example, the RGB color space, of the positive slopefor the color variation caused by the illumination change across thesingle material. BIDR normal information can be used to defineconstraints between, for example, Type C tokens, as a basis forsegregating the illumination and material reflectance components of animage, as will be described.

According to a feature of the present invention, BIDR normal informationis identified for each pixel in an image depicted in an image file 18.U. S. Patent Publication US 2014/0050395 teaches various techniques toidentify spatially varying BIDR normals, for use in an image process.One technique taught in U. S. Patent Publication US 2014/0050395calculates a BIDR normal information value for each pixel of the imagefile 18. Each normal information value is associated with correspondingpixel positions for the pixels used to calculate the respective normal,N, for example, from among pixels p(1, 1) to p(n, m) of an image file 18being processed, as shown in FIG. 2. For example, a user can enter a setof lit-dark pairs that each include a patch of lit pixels and acorresponding patch of dark pixels, from a same material (block 1022).In an exemplary embodiment of the present invention, a user designates,for example, via a touch screen action, one or more sets of lit-darkpairs of pixel blocks, the pairs each corresponding to lit and shadowedregions of a same material, respectively, depicted in the image of theimage file 18 being processed. Each pixel block includes, for example,an n×n array of pixels.

According to a feature of the present invention, the CPU 12 is operatedto calculate a BIDR normal information value for each pixel of an imagefile, and create a BIDR normal image, with each pixel location, asshown, for example, in FIG. 2, indicating a respective BIDR normalinformation value, as calculated by the CPU 12 as a function of theuser-selected lit-dark pairs, according to the teachings of U. S. PatentPublication US 2014/0050395 (block 1024). In alternative exemplaryembodiments of the present invention, BIDR normal information values canbe calculated based upon an automatic calculations, as also taught in U.S. Patent Publication US 2014/0050395.

In step 1026, the CPU 12 operates to create a scale-spaced pyramid ofthe BIDR normal representation of the image, by mapping the BIDR normalinformation values from one resolution to another. In the creation ofthe scale-spaced pyramid for the image file 18, as described above (step1012), there is a mapping of pixels, from one resolution to the nextresolution of the pyramid, as for example, via the color purity routineof FIG. 27. The mapping is used to map a pixel in a lower resolution toa corresponding pixel, in the next higher resolution of the pyramid.

According to a feature of the present invention, another grouping of theimage is created when the image file 18 is a frame of a video recording.In the case of a video, the CPU 12 operates to calculate temporalinformation relevant to pixel locations. The temporal information canalso be used to define constraints as a basis for segregating theillumination and material reflectance components of an image, as will bedescribed. In an exemplary embodiment of the present invention, for thetemporal information, the CPU 12 computes color flow vectors as afunction of Type C tokens identified in the image file 18 beingprocessed, and an image file 18′ that is the previous frame of the video(block 1028). FIG. 3g is a flow chart for the color flow vector step ofFIG. 3 c.

In some videos, the camera used to record the scene, and the sceneitself, are static, or depict slow moving objects. In other videos, thecamera and/or objects in the scene undergo rapid movement. When thevideo depicts rapid motion, a rough estimate of motion is required forfurther processing. A known optical flow algorithm can be executed bythe CPU 12 to provide the rough estimate of motion. In step 1400, a usercan select whether the CPU 12 executes an optical flow algorithm. In analternative exemplary embodiment, an automatic selection of optical flowcan be made based upon an initial estimate of scene motion. If yes, theCPU 12 executes the optical flow algorithm to calculate optical flowinformation between current and previous frames of the video (k and k−1,corresponding to the image file 18 and the image file 18′, respectively)(block 1402).

In step 1404, the CPU 12 uses the rough estimate of the optical flowinformation to ascertain the approximate location of a selected Type Ctoken Tk, from image file 18 (frame k), in the image file 18′ (framek−1). In step 1406, the CPU 12 shifts a center of a search field withinthe previous frame k−1 to the location indicated by the optical flowinformation.

When the user selects not to use the optical flow information, as in thecase of a video with static or slow moving objects, the CPU 12 executesstep 1408. In step 1408, the CPU 12 centers the search location in framek−1 based upon the location of the token Tk in the frame k. The use ofthe same location should be accurate due to the slow moving objects orstatic conditions of the video.

In step 1410, the CPU 12 locates token Tk−1, the token in the searchlocation of frame k−1, that is the closest match to token Tk. In step1412, the CPU 12 compares the color difference between Tk and Tk−1 to athreshold value. The threshold is set to a value, for example,comparable to the noise variation of the equipment used to record thevideo. If the color difference is not less than the threshold, the CPU12 does not assign any color flow vector to the token (step 1414). Ifthe color difference is less than the threshold, the CPU 12 proceeds tostep 1416.

In step 1416, due to the close similarity of color values between thetokens, Tk and Tk−1, as established in step 1412, the CPU 12 computes acolor flow vector for the movement of the token, from frame k−1 to framek. The color flow vector is expressed as the difference in x,y positionsof the centroid of the tokens Tk and Tk−1, relative to an x,y frame ofreference placed upon the P(1, 1) to P(N, M) pixel array, as shown inFIG. 2. Upon completion of step 1416, the CPU 12 assigns the computedcolor vector for the current Tk, Tk−1 pair (step 1418).

Referring once again to FIG. 3c , to complete step 1028, the CPU 12executes the routine of FIG. 3g for each token of the frame k, resultingin a listing of all tokens assigned a color flow vector and thecorresponding color flow vectors. The CPU 12 then creates an imagerepresentation, with a computed color flow vector at each pixelcorresponding to the centroid of each token analyzed and assigned acolor flow vector during the execution of the routine of FIG. 3 g.

In step 1030, the CPU 12 operates to create a scale-spaced pyramid ofthe color flow vector representation of the image, by mapping the colorflow vectors from one resolution to another, for example, according tothe mapping from FIG. 27. In the case of color flow vectors, the valueof the color vector mapped from a higher resolution is divided by theamount of reduction in resolution, from one level of the pyramid to thenext, lower resolution level. For example, if D pixels of a higherresolution level correspond to one pixel at the lower resolution of thenext level, the color flow vector value is divided by a factor of 1/D.Again, the pixel mapping can be implemented according to the colorpurity routine of FIG. 27, as will be described.

According to a feature of the present invention, a user can indicatevarious parameters and/or information relevant to the image process. Forexample, a user can enter a proxy scale (block 1032). The proxy scalecan limit the processing to a certain resolution, less than the fullresolution level of the pyramid. This can be used to increase the speedof execution, particularly when the original image is large. The proxysetting can be used to obtain fast results as a preview, before fullprocessing. A user can also specify parameters relevant to thetokenization process (block 1034). In addition, a user can, via a touchscreen action, draw scribble lines on a display of the image. Thescribbles can include, for example, a fully lit scribble to indicate anarea of the image that is fully illuminated, a same reflectancescribble, to indicate an image region of a same material, and a sameillumination scribble, to indicate a region of the image that is at thesame level of illumination. The scribbles supply accurate informationfor the CPU 12 to impose on constraints.

In steps 1038 a to 1038 d, the CPU 12 operates to create constraintsbased upon the set of pyramids created in steps 1016, 1020, 1026 and1030. The constraints are concatenated in a matrix array, for example, amatrix equation of the form [A] [x]=[b], as shown, for example, in FIGS.21 and 22 (block 1040). The CPU 12 operates to solve the matrix equation(block 1042), and after performing optional post processing, as will bedescribed in more detail (block 1044), outputs each of a materialreflectance intrinsic image (1046) and an illumination intrinsic image(1048). The material reflectance image captures the reflectanceproperties of surfaces depicted in the image (the percentage of eachwavelength of light a surface reflects). The illumination image capturesthe intensity and color of light incident upon each point on thesurfaces depicted in the image.

All of the constraints of steps 1038 a to 1038 d, and the arrangement ofthe matrix equation [A] [x]=[b], are computed pursuant to aspatio-spectral operator/constraint/solver model for an imagesegregation, according to the present invention. As shown in FIG. 3a ,in step 1004 the image file 18, and/or any selected one or more of thealternative representations image files 18 a-n, are subject to an imagesegregation. As shown in FIG. 3c , multiple other groupings of theimage, for example, blend pixels, BIDR normals and color flow vectors,can be created by the CPU 12 to supplement the image segregation.

A fundamental observation underlying a basic discovery of the presentinvention, is that an image comprises two components, materialreflectance and illumination. All changes in an image are caused by oneor the other of these components. Spatio-spectral information isinformation relevant to contiguous pixels of an image depicted in animage file 18, such as spectral relationships among contiguous pixels,in terms of color bands, for example RGB values of the pixels, and thespatial extent of the pixel spectral characteristics relevant to acharacteristic of the image, such as, for example, a single materialdepicted in the image or illumination effecting the image. The BIDRmodel discussed above is an example of spatio-spectral information,predicting color change among contiguous pixels across a singlematerial.

As recognized by the BIDR model, the illumination includes an incidentilluminant and an ambient illuminant. The spectra for the incidentilluminant and the ambient illuminant can be different from one another.Thus, a spectral shift is caused by a shadow, i.e., a decrease of theintensity of the incident illuminant. The spectral shift can cause avariance in color of material depicted in the scene, from full shadow,through the shadow penumbra, to fully lit. Pursuant to a feature of thepresent invention, spectral shift phenomena is captured inspatio-spectral information, such as, for example a BIDR normal. Thespatio-spectral information includes a spectral ratio: a ratio basedupon a difference in color or intensities between two areas of a scenedepicted in an image, which may be caused by different materials (anobject edge), an illumination change (illumination boundary) or both.

According to a further feature of the present invention, spatio-spectraloperators are generated to facilitate the image segregation processperformed in step 1004, for the segregation of illumination and materialaspects of a scene depicted in the retrieved image file 18, and anyother representations of the image, such as image file 18 a.Spatio-spectral operators comprise representations or characteristics ofan image that encompass spatio-spectral information usable in theprocessing of material and illumination aspects of an image. Thespatio-spectral operators are subject to constraints that defineconstraining spatio-spectral relationships between the operators, forinput to a solver. The operators can be identified at any one of theselectively varied representations of the original image, and theconstraints defined in terms of operators within a single representationor between operators in different representations.

A constraint can be imposed among the different levels of the pyramid,for example, to improve and expedite segregation at the finer scale(image file 18). To that end, as described above, a segregation intoillumination and material is first quickly and efficiently performed bythe CPU 12 (according to the spatio-spectral operator/constraint/solvermodel, described in detail below) for the relatively coarse image (forexample, image file 18 a). Thereafter, for more consistent and accurateresults, during segregation at the finer scale, material color at eachlocal area of the relatively finer scale image (for the material imagegenerated for image file 18) is constrained such that a weighted averageof the color at a local area of the fine scale equals the color of acorresponding local area of the coarse scale.

Such a weighted average constraint can be expressed, in a simplifiedanalysis for a three pixel local area of the fine scale representation,as follows: if the log intensities are related as:i2(x,y)=0.5*i1(x,y)+0.25*i1 (x−1,y)+0.25*i1 (x+1,y), wherein i2(corresponding, for example, to the log of image file 18 a) is a blurredversion of i1 (corresponding to the log of image file 18); then the logmaterial values are related as: m2(x,y)=0.5*m1 (x,y)+0.25*m1(x−1,y)+0.25*m1 (x+1,y), wherein m1 and m2 are the material componentsof i1 and i2, respectively. Typically, the analysis is performed for alocal area of the fine scale that is larger than the area correspondingto the pixel of the coarse scale representation, to avoid aliasing. Forexample, in pyramid wherein the coarse scale has half the pixels of thefine scale, a 5×5 pixel square would be used to calculate the weightedaverage for the color at the fine scale, and then constrained to be thecolor of a single pixel in the coarse scale representation.

In the exemplary embodiment of the present invention, the solverincludes a mathematical processing engine that operates to obtain anoptimized solution for the generation of an intrinsic image, such as amaterial image and/or an illumination image derived from the originalimage stored in the retrieved image file 18 or any other representationof the image, as a function of the constraining relationships betweenthe spatio-spectral operators.

Spatio-spectral operators include, for example, tokens, token mapinformation, log chromaticity representation values, X-junctions, BIDRmodel representations, a boundary representation, and a texton histogrambased pixel representation. Each of the spatio-spectral operators can becomputed and applied at the image file 18, and/or at any of the imagefiles 18 a, . . . 18 n, and any solution at any level can be related toany solution at any other level each via the upsampling and downsamplingrelationships described above.

Pursuant to a feature of the present invention, a token is a connectedregion of an image wherein the pixels of the region are related to oneanother in a manner relevant to identification of image features andcharacteristics such as identification of materials and illumination.The use of tokens recognizes the fact that a particular set ofmaterial/illumination/geometric characteristics of an image extendsbeyond a single pixel, and therefore, while the image processingdescribed herein can be done on a pixel level, tokens expedite a moreefficient processing of image properties. The pixels of a token can berelated in terms of either homogeneous factors, such as, for example,close correlation of color values among the pixels, or nonhomogeneousfactors, such as, for example, differing color values relatedgeometrically in a color space such as RGB space, commonly referred toas a texture.

Exemplary embodiments of the present invention provide methods andsystems to identify various types of homogeneous or nonhomogeneoustokens for improved processing of image files. The present inventionutilizes spatio-spectral information relevant to contiguous pixels of animage depicted in an image file 18, or contiguous pixels of anyselectively varied representation (image files 18 a, . . . 18 n) toidentify token regions within the original image and/or within any ofthe variations thereof. Such tokens identified in different selectivelyvaried representations of the image can be used in, for example, theweighted average color constraint discussed above.

According to one exemplary embodiment of the present invention,homogeneous tokens are each classified as either a Type A token, a TypeB token or a Type C token. A Type A token is a connected image regioncomprising contiguous pixels that represent the largest possible regionof the image encompassing a single material in the scene. A Type B tokenis a connected image region comprising contiguous pixels that representa region of the image encompassing a single material in the scene,though not necessarily the maximal region corresponding to thatmaterial. A Type C token comprises a connected image region of similarimage properties among the contiguous pixels of the token, for example,similar color and intensity, where similarity is defined with respect toa noise model for the imaging system used to record the image. Theweighted average constraint discussed above, can be implemented at aType C token level, constraining the weighted average color of atoken(s) at the fine scale to the color of a corresponding token(s) at acoarse scale.

A linear token is a nonhomogeneous token comprising a connected regionof the image wherein adjacent pixels of the region have differing colormeasurement values that fall within a cylinder in RGB space, from a darkend (in shadow) to a bright end (lit end), along a positive slope. Thecylinder configuration is predicted by the BIDR model, according to afeature of the present invention, when the color change is due to anillumination change forming a shadow (i.e. a decrease in the intensityof the incident illuminant as the interplay between the incident ordirect illuminant and the ambient illuminant in the illumination field)over a single material of a scene depicted in the image.

For purposes of describing, identifying and using linear tokens, theBIDR model can be stated as:I_((x, y, z, φ, λ))=c_(b)(λ)l_(d)(λ)γ_(b)+M_(a)(λ)c_(b)(λ), where:I_((x, y, z, θ, φ, λ)), is the radiance of a surface point at (x, y, z)in the direction θ, φ for the wavelength λ, c_(b)(λ) is the geometryindependent body reflectance of a surface for the wavelength λ, l_(d)(λ)is the incident illuminant for the wavelength λ. γ_(b) is the product ofa shadow factor s_(x, y, z) and a geometric factor m_(b) (θ_(i)), andM_(a)(λ) is the integral of the ambient illuminant and geometric bodyreflectance over a hemisphere, excluding the incident illuminant. Formore detailed information on the BIDR model, reference should be made toU.S. application Ser. No. 11/341,751, filed Jan. 27, 2006, entitled:“Bi-illuminant Dichromatic Reflection Model For Image Manipulation,”published as US 2007/0176940 on Aug. 2, 2007.

Token map information indicates locations of tokens within an image,relative to one another. The map information is used to identifyneighboring tokens for performing an analysis of token neighborrelationships relevant to constraining spatio-spectral relationshipsbetween tokens, for input to the solver. A token map can be generatedfor each selectively varied representation of the original image inrespect of tokens identified at each respective variation.

Log chromaticity representation values provide illumination invariantvalues for pixels of the image. Logarithmic values of the color bandvalues of the image pixels are plotted on a log-color space graph. Thelogarithmic values are then projected to a log-chromaticity projectionplane oriented as a function of the BIDR model. The chromaticity planevalues are substituted for the color band values (for example, RGBvalues) of each pixel. For more detailed information on log chromaticityrepresentation values, reference should be made to U.S. Pat. No.7,596,266.

An X-junction is an area of an image where a material edge and anillumination boundary cross one another. An X-junction is an optimallocation for an accurate analysis of material and illumination aspectsof an image.

A boundary representation is an arrangement of pixels, on each side of aboundary, formed by, for example, adjacent Type B tokens. Thearrangement is used to facilitate an analysis of the boundary toclassify the boundary as a material boundary on a smooth surface (asopposed to another type of boundary, for example, an illumination edge,depth boundary or simultaneous illumination and material change). Thepixel representation is configured to provide samples of pixels withineach of the Type B tokens forming the boundary. The pixels of thesamples are subject to spatio-spectral analysis, and the results arecompared to determine the likelihood that the respective boundarycorresponds to a material change.

A texton is a homogeneous representation for a region of an image thatcomprises a texture. Image texture can be defined as a function ofspatial variation in pixel intensities. Image texture patterns arefrequently the result of physical or reflective properties of the imagesurface. Commonly, an image texture is associated with spatialhomogeneity and typically includes repeated structures, often with somerandom variation (e.g., random positions, orientations or colors). Imagetextures are also often characterized by certain visual properties suchas regularity, coarseness, contrast and directionality. An example ofimage texture is the image of a zebra skin surface as it appears to bespatially homogenous and seems to contain variations of colorintensities which form certain repeated patterns. Some image texturescan be defined by geometric characteristics, such as stripes or spots. Atexton based operator, such as a texture token, as will be described,transforms patterns of differing reflectance caused by a texturedmaterial into a homogeneous representation that captures the spectraland spatial characteristics of the textured region in the image.

Constraints between spatio-spectral operators comprise, for example, ananchor constraint, a same illumination constraint, a smooth illuminationconstraint, a Type B token or same material constraint, a Linear tokenconstraint, a BIDR enforcement constraint, a same texton histogramconstraint, a log chromaticity similarity constraint, an X junctionconstraint, and a boundary representation constraint. Other constraintscan be formed as between spatio-spectral operators, or other indicia,from different ones of the selectively varied representations of theimage, such as, for example, the weighted average color constraintdescribed above. Each constraint is configured as a constraint generatorsoftware module that defines the spatio-spectral operators utilized bythe respective constraint and provides an expression of the constrainingrelationship imposed upon the constituent operators.

An anchor constraint utilizes, for example, a number ofbrightest/largest Type C tokens in an image. The constrainingrelationship is that the material of the selected brightest/largest TypeC tokens is constrained to be an absolute value for the color/brightnessobserved in the image. The constraint anchors a material map for theimage at an absolute brightness to avoid relative brightnessconstraints.

A same illumination constraint utilizes Type C tokens and Type B tokensidentified in an image and token map information. The constrainingrelationship is that adjacent Type C tokens, as indicted by the tokenmap information, are at the same illumination, unless the adjacent TypeC tokens are part of the same Type B token. The term “same” inconnection with the term “illumination” is used to mean an average valuewith respect to a noise model for the imaging system used to record theimage. This constrains any observed differences in appearance betweenadjacent Type C tokens, that are not part of the same Type B token, tobe a material change, as will appear.

A smooth illumination constraint is similar to the same illuminationconstraint. However, rather than constraining all pixels of adjacentType C tokens to be of the same illumination, as in the sameillumination constraint, in the smooth illumination constraint, theconstraint is based upon the average illumination of the pixels near ashared boundary between adjacent Type C tokens. This constrains theillumination field to be somewhat smooth, as opposed to piecewiseconstant (the same, as defined above) throughout a token.

A Type B token or same material constraint also utilizes Type C tokensand Type B tokens. However, the constraining relationship is that allType C tokens that are part of the same Type B token are constrained tobe of the same material. This constraint enforces the definition of aType B token, that is, a connected image region comprising contiguouspixels that represent a region of the image encompassing a singlematerial in the scene, though not necessarily the maximal regioncorresponding to that material. Thus, all Type C tokens that lie withinthe same Type B token are by the definition imposed upon Type B tokens,of the same material, though not necessarily of the same illumination.The Type C tokens are therefore constrained to correspond to observeddifferences in appearance that are caused by varying illumination.

Accordingly, the Type B token constraint is complementary to the sameand smooth illumination constraints, which, as opposed to illuminationchange, constrain observed differences to correspond to material change,as described above. This is due to the fact that in each of the same andsmooth illumination constraints, Type C tokens that are adjacent and notpart of the same Type B token, are constrained to the same illumination.These Type C tokens should comprise different materials, since by theconstraint, they are not in the same Type B token and therefore, by thedefinition of Type B tokens enforced by the constraint, do not encompassa single material, so illumination should be a constant, and anyobserved difference is considered as attributable to a material change.

To summarize, pursuant to a feature of the present invention, the Type Cand Type B token spatio-spectral operators are defined to providecharacteristics of an image that enable segregation of illumination andmaterial. Type C tokens each comprise a connected image region ofsimilar image properties, for example similar color, as recorded andstored in an image file 18. Thus, adjacent Type C tokens indicate someform of change in the image or else they would form the same Type Ctoken. Type B tokens encompass a single material. The complementaryconstraints of the same/smooth illumination constraints and the Type Btoken constraint enforce relationships between the tokens that indicateeither a material change or an illumination change.

If the adjacent Type C tokens are within the same type B token, as inthe Type B token constraint, the differences between them shouldcorrespond to illumination change due to the same material property ofthe common Type B token. If the adjacent Type C tokens are not withinthe same Type B token, as in the same/smooth illumination constraints,the difference between them should then correspond to a material changesince they are not both defined by a common, single material Type Btoken.

A Linear token constraint utilizes Type C tokens and Linear tokens. Theconstraining relationship is that a difference between two Type Ctokens, spaced by a Linear token, approximately equals a characteristicilluminant spectral ratio for the image. As defined, a Linear tokenfollows a cylinder configuration along a positive slope, through colorspace. The BIDR model predicts that the positive slope equals acharacteristic illuminant spectral ratio for the image. Thus, the colordifference between two Type C tokens, one at each of the dark end andbright end of a Linear token, should reflect the value of the respectivecharacteristic illuminant spectral ratio for the image.

A BIDR enforcement constraint utilizes Type C tokens and a BIDR modeldefined normal vector for the log-chromaticity projection plane. Theconstraining relationship is that the illumination for all Type C tokensin a local patch of the image forms a set of parallel lines in log-colorspace, the orientation of the parallel lines being defined by the BIDRmodel defined normal vector. The constraint therefore enforces theillumination field present in the image to explicitly fit the BIDR modelprediction for the illumination.

Thus, each of the Linear token constraint and the BIDR enforcementconstraint utilize BIDR model predictions as a basis to segregateillumination and material aspects of an image. The BIDR model predicts acolor change in an image when the color change is due to an illuminationchange forming a shadow (i.e. a decrease in the intensity of theincident illuminant as the interplay between the incident or directilluminant and the ambient illuminant in the illumination field) over asingle material of a scene depicted in the image. The color changeprediction of the BIDR model accurately constrains all color bandvariations among Type C tokens to illumination field effects occurringin an image by operating as a function of the interplay between thespectral variations occurring between incident illuminant and ambientilluminant components of the illumination field. Thus, BIDR model basedconstraints couple all color band variations into one integralconstraining relationship.

A same texton histogram constraint utilizes Type C tokens and textonhistogram operators identified for texture regions, such as texturetokens, within an image. A texton analysis is utilized wherein eachpixel of the image (or pixels of those regions of an image identified ascomprising a texture) from the recorded color band representation of therespective image file 18, such as, for example, RGB color band values,is converted to a two band representation wherein the two bands comprisea texton label and a texton histogram label. The two bandrepresentations are then used to identify texture tokens, as will bedescribed below. A constraint can be imposed that all Type C tokenswithin the same texture token are of the same mean material.

A log chromaticity similarity constraint utilizes Type C tokens and logchromaticity representation values. The constraining relationship isthat those Type C tokens having pixels with similar log chromaticityrepresentation values are constrained to a same color value, withobserved differences being attributed to variations in the illuminationfield.

An X-junction constraint utilizes Type C tokens and X-junctionoperators. As noted above, an X-junction is an area of an image where amaterial edge and an illumination boundary cross one another.X-junctions are typically identified by four Type C tokens, two pairs ofsame material Type C tokens forming the material edge, with each samematerial pair including an illumination boundary dividing the respectivesame material into lit and shadowed pairs of Type C tokens. Theconstraining relationship: 1) a Type B token constraint is imposedbetween each same material pair of Type C tokens forming the X-junction(those with an illumination boundary between them), and 2) a sameillumination constraint is imposed between each pair of Type C tokensforming the material edge of the X-junction. For a more detaileddescription of X-junctions and the relationships of constituent tokens,reference should be made to U.S. application Ser. No. 11/341,742, filedJan. 27, 2006, entitled: “Method And System For Identifying IlluminationFlux In An Image,” published as US 2006/0177149 on Aug. 10, 2006.

A boundary representation constraint is defined by a standard ratioconstraint. An analysis performed on a boundary representation, whenindicating a material change, provides an estimate of the ratio ofcolors between two adjacent regions defined by the boundary, forexample, the adjacent Type B tokens, even when the illumination variesover the regions. The constraint states that the ratio of the colors oftwo adjacent regions is X. The boundary representation analysis isexecuted at the level of Type B tokens, to classify a boundary as beingcaused by a material change, then propagated down to the level of theconstituent Type C tokens. For a more detailed description of a boundaryanalysis, at the Type B token level, reference should be made to U.S.application Ser. No. 12/079,878, filed Mar. 28, 2008, entitled “Systemand Method For Illumination Invariant Image Segmentation.”

According to a feature of the present invention, the boundaryrepresentation constraint states that all adjacent pairs of Type Ctokens along the boundary, (one Type C token on each side of theboundary, and all of the Type C tokens being within the Type B tokensforming the respective boundary), have colors that satisfy the ratio X,as indicated by the boundary representation analysis.

According to a preferred embodiment of the present invention, each ofthe above described constraints can be classified into one of threebasic types of constraints, an absolute material color constraint, asame material constraint and a relative reflectance constraint. Theabsolute material constraint constrains the material at a particularlocation of an image to be a certain color, as implemented in, forexample, the anchor constraint. The same material constraint constrainsoperators relevant to an image (for example, two pixels or Type Ctokens) to be of the same material. The same material type of constraintcan be implemented in, for example, Type B, X-junction, log chromaticitysimilarity, same texton histogram and linear token constraints. Therelative reflectance constraint constrains operators relevant to animage (for example, two pixels or Type C tokens) to have a similarity ofreflectance characteristics, such as defined by smooth illumination andsame illumination constraints, and which can be specified by X-junction,and boundary representation constraints.

In another example of the selectively varied representation embodimentof the present invention, such as the multi-resolution embodiment,groupings provided by the spatio-spectral operators, such as Type Ctokens, Type B tokens and texture tokens, are identified at one scale,such as a fine scale (for example image file 18) and constrainingrelationships among those groupings are projected to a coarse scale (forexample image file 18 a). In this exemplary embodiment of the presentinvention, a constraint implemented at a fine scale representation, suchas, for example, the same material constraint of a Type B tokenconstraint, is arranged to constrain pixels or Type C tokens to bewithin in the same Type B token, and further, to be within a relativelylimited neighborhood of the fine scale representation. The neighborhoodis limited to an extent somewhat less than the extent of a typicalrelevant Type B token, for example, a neighborhood comprising a 50×50pixel array. The limited neighborhood restriction has the advantage ofassuring a sparse matrix for an accurate solution, with operatingefficiencies for speedy processing, resulting from the limited extent ofthe image subject to analysis according to the constraint. However, theimplementation across multiple scales also assures a solution accordingto a long range constraint since a short range within a coarse scalerepresentation corresponds to a relatively long range in the fine scalerepresentation.

A projection of the constraining relationship imposed on a grouping,from one scale, to another scale, is implemented, for example, byupsampling the grouping identified and used in the constraint at thefine scale representation, to a next coarse scale representation. Theperformance of the constraint is repeated at the coarse scale, againlimiting the neighborhood processed at the coarse scale, and subject tothe inter-scale relationship defined by the filtering/downsamplingoperation, and so on throughout the various scales of the pyramid. Thelimit at the coarse scale is a multiple of the same limit of therelatively fine scale. For example, as shown in FIG. 3b , a singlepixel, P_(a)(3, 2) in the relatively coarse resolution image file 18 acorresponds to pixels P(5, 3), P(6, 3), P(5, 4) and P(6, 4) of fineresolution image file 18. The grouping can be identified at any of thescales of resolution, and projected via upsampling or downsampling tothe other scales.

An exemplary solver according to a feature of the present inventioncomprises a mathematical processing engine for executing an optimizingfunction, for example, optimization of results in an equation expressedby: [A] [x]=[b], where [A] is a matrix of values that are to besatisfied by (and therefore, taken as solved for by) the definitions ofthe operator(s) and the constraining relationship(s) for theoperator(s), as indicated by selected constraint(s), [x] is a matrix ofvariables for which the equation is finding an optimal solution, forexample, one of an illumination or material component of an imagecomponent, for example, a pixel or token, and [b] is a matrix of valuesobserved in an image selected for processing, for example, the recordedvalues for the RGB color bands of each pixel of an image file 18 orcolor values of pixels within a selectively varied representation of theoriginal image depicted in image file 18 (e.g. image files 18 a . . . 18n). The optimizing equation can be implemented in a mathematicaloptimizing function selected from a set of known optimization solverssuch as, for example, known convex optimization operations such as aleast squares solver, or a preconditioned conjugate gradient solver.

According to the selectively varied representation embodiment of thepresent invention, a solution according to the [A] [x]=[b] matrixequation can be accomplished at each representation, for example, ateach scale of the scale-spaced pyramid, either simultaneously orsequentially. The inter-scale relationship constrains the multi-scalesolution, while introducing efficiencies in operation execution forincreased speed. By dividing processing across different scales,solutions are found for a set of smaller systems rather than one largecomplex system. Thus, the solutions are more accurate and the CPU 12 canexecute in faster computation times while utilizing substantially lessmemory.

FIG. 4 shows a functional block diagram of an image segregation systemarchitecture, implemented in, for example, the computer system of FIG.1, according to a feature of the present invention. Alternatively, thefunctional blocks of FIG. 4 can be implemented in a dedicated hardwarecircuit arranged to perform the functionality of the blocks of FIG. 4.An image 32 (as depicted in an image file 18, or one of the selectivelyvaried representations in image files 18 a, . . . 18 n) is input to apreprocessing block 33. The preprocessing block 33 can perform suchfunctions as correction of chromatic aberration in the image 32,combining multiple images to provide a high dynamic range image,linearize pixel data for the image, and so on, for an image optimizedfor processing. The pre-processed image is then input to a Type Ctokenization block 35 which operates to identify Type C tokens in thepre-processed image, in the manner described below with reference toFIG. 6a . Type C tokens are common to many of the constraints utilizedin exemplary embodiments of the present invention, thus, an initialidentification of Type C tokens for an input image 32 expedites furtherprocessing.

In an exemplary embodiment of the present invention, the CPU 12 executescode to implement both the preprocessing block 33 and the Type Ctokenization block 35, as well as a service provider 24, that functionsas a central agent and caching structure (configured in the memory 16),to handle an image for processing according to the teachings of thepresent invention. The service provider 24 receives and stores thepre-processed image and related Type C token information from the Type Ctokenization block 35, and is coupled to an operators block 28 (executedby the CPU 12) arranged to generate any other operators for the imagerequired by selected constraints, as will appear. The service provider24 is also coupled to a global features extraction input 29. The globalfeatures extraction input 29 can be used to provide the system withinformation relevant to an image being processed, such as an indicationof light source when the image was taken (sunlight, fluorescent light,incandescent light), time of day, location, domain knowledge, such asinformation relevant to the nature of the image, such as interior,exterior, buildings, lawns with green grass, trees with leaves in bloom,etc., and any other parameters relevant to image processing. The serviceprovider 24 stores the global features extraction input 29 with arelated input image 32.

A constraint builder 26 is coupled to the service provider 24. Theconstraint builder 26 uses a constraint generator library (configuredwithin the memory 16) that stores the constraint generator softwaremodules for the various constraints described above. The serviceprovider 24 and constraint builder 26 operate to arrange spatio-spectraloperators relevant to the pre-processed image, according to selectedones of the constraint generator software modules, in for example, the[A] [x]=[b] matrix equation.

A solver 30 (executed by the CPU 12) is coupled to the constraintbuilder 26, and implements an optimization operation, as describedabove, for an optimal solution for the [A] [x]=[b] matrix equation, foruse in generating intrinsic images from the pre-processed image. Thesolver 30 is also coupled to a post-processing block 36 (executed by theCPU 12) for certain post-processing operations. The post-processingoperations can include, for example, monotonicity maintenance. Inmonotonicity maintenance, if two large regions exhibit a lineartransition in the input image 32, the transition should remain a lineartransition in the output intrinsic image 34. Post-processing can alsoinclude illumination propagation, that serves to fill in holes left bythe solver 30, illumination-map based white balancing and otherfiltering, smoothing processes. The post-processing block 36 outputsintrinsic images 34.

Referring now to FIG. 5, there is shown a graphical user interface (GUI)for use in connection with an exemplary implementation of the imagesegregation system architecture feature of the present invention. TheGUI of FIG. 5 is displayed on the monitor 20 of the computer system 10by the service provider 24 for a user to select a desired imagesegregation operation. The upper left hand corner of the GUI indicatesOpen Image, Crop Image, Show Parameters, Segregate and Multi-Resolutionselection indicators. A user can move and click a cursor on a desiredselector indicator. The Open Image indicator lists all image files 18currently stored in the memory 16 and enables the user to select animage for processing. The selected image is input 32 (see FIG. 4) to theservice provider 24 (via the preprocessing block 33 and the Type Ctokenization block 35) which operates to display the selected image atthe upper center of the monitor 20 (FIG. 5).

A material image derived by operation of the exemplary segregationsystem from the selected image is output 34 (see FIG. 4) after executionof the image segregation processing by the solver 30 and displayed atthe lower right hand of the monitor 20 (FIG. 5). The derivedillumination image is displayed at the lower right hand of the monitor20 (FIG. 5).

According to a feature of the present invention, the Crop Image selectorpermits a user to crop a selected image so as to process a portion ofthe overall image. The Show Parameter selector displays parametersrelated to the selected image file 18. Parameters for each image file 18can be stored in a parameter data file associated with a correspondingimage file 18, and include any parameters relevant to the processing ofthe image depicted in the associated image file 18, for example theglobal features extraction input 29. Parameters can include any datarelevant to image processing such as, for example, any variable forpixel analysis by the CPU 12, as for example, in the generation ofspatio-spectral operators, and domain knowledge, such as informationrelevant to the nature of the image, such as interior, exterior,buildings, lawns with green grass, trees with leaves in bloom, etc.

A multi-resolution selector can display a drop down menu that providesoptions such as either a Gaussian or Laplacian pyramid generation, withspecification of the number of scales, and the resolution of each scale.The drop down menu can also include selections for a computer operationfor illumination and material image segregation, to be performed at arelatively coarse scale representation of the image, and then used toconstrain a solution at a relatively finer scale resolution. In anotherselection option, a segregation of illumination and material aspects ofthe image is selectively divided among representations at differingscales of image resolution, as described above.

Below the selection indicators is a list of each of the optimizingfunctions that can be used as the solver 30, and a further list of eachof the constraint generators contained in the constraint generatorlibrary of the constraint builder 26. A user selects a desiredmathematical operation and one or more of the constraints to be imposedupon the selected image. After selection of the image to be processed,the constraints to be imposed and the mathematical operation to beexecuted, the user can click on the Segregate indicator to commenceimage segregation processing.

Upon commencement of the image segregation processing, the serviceprovider 24 generates a scale-spaced pyramid, if a multi-resolutionoperation is selected, and retrieves the constraint generator softwaremodules for the selected constraints to identify the spatio-spectraloperators utilized by the selected constraints. Any spatio-spectraloperators not already stored by the service provider 24 are generated bythe operators block 28, for the image being segregated, and the serviceprovider 24 caches the results. The cached results can be reused in anysubsequent operation for a selected image, with the same set ofassociated parameters.

In a multi-resolution operation, the service provider 24 can sequenceoperations, such as perform the segregation at the relatively coarserepresentation of image file 18 a, store the results, then proceed to asegregation of image file 18, enforcing the color constraint describedabove. The service provider 24 (operating via the CPU 12) can also beoperated to execute a standard edge detector, then define regions of therelatively fine resolution image of image file 18 around each identifiededge. An image segregation is then performed in respect of theidentified edge regions of the image file 18. A subsequent imagesegregation is performed on relatively coarse image file 18 a forregions corresponding to all non-edge regions of image file 18.

In an example of an implementation of the spatio-spectraloperator/constraint/solver model of the present invention, when theselected constraint is a same illumination constraint, the serviceprovider 24 identifies Type C tokens, Type B tokens and a token map forthe selected image. The Type C tokens were generated by the Type Ctokenization block 35. The service provider 24 operates the operatorsblock 28 to generate the remaining operators specified by the sameillumination constraint.

Referring now to FIG. 6a , there is shown a flow chart for generatingType C token regions in the image file of FIG. 2, (as depicted in one ormore or each of image file 18, and image files 18 a, . . . n), accordingto a feature of the present invention. Type C tokens can be readilyidentified in an image by the Type C tokenization block 35, utilizingthe steps of FIG. 6a . The operators block 28 can then analyze andprocess the Type C tokens to construct Type B tokens when specified by aselected constraint, as will appear.

A 1st order uniform, homogeneous Type C token comprises a single robustcolor measurement among contiguous pixels of the image. At the start ofthe identification routine of FIG. 6a , the CPU 12 (executing as theType C tokenization block 35) sets up a region map in memory. In step100, the CPU 12 clears the region map and assigns a region ID, which isinitially set at 1. An iteration for the routine, corresponding to apixel number, is set at i=0, and a number for an N×N pixel array, foruse as a seed to determine the token, is set an initial value,N=N_(start). N_(start) can be any integer>0, for example it can be setat set at 11 or 15 pixels.

At step 102, a seed test is begun. The CPU 12 selects a first pixel,i=(1, 1) for example (see FIG. 2), the pixel at the upper left corner ofa first N×N sample of the image file 18. The pixel is then tested indecision block 104 to determine if the selected pixel is part of a goodseed. The test can comprise a comparison of the color value of theselected pixel to the color values of a preselected number of itsneighboring pixels as the seed, for example, the N×N array. The colorvalues comparison can be with respect to multiple color band values (RGBin our example) of the pixel or the filter output intensity histogramrepresentation of the pixel, in the event the image was filtered fortexture regions, as described above. If the comparison does not resultin approximately equal values (for example, within the noise levels ofthe recording device for RGB values) for the pixels in the seed, the CPU12 increments the value of i (step 106), for example, i=(1, 2), for anext N×N seed sample, and then tests to determine if i=i_(max) (decisionblock 108).

If the pixel value is at i_(max), a value selected as a threshold fordeciding to reduce the seed size for improved results, the seed size, N,is reduced (step 110), for example, from N=15 to N=12. In an exemplaryembodiment of the present invention, i_(max) can be set at i=(n, m). Inthis manner, the routine of FIG. 5a parses the entire image at a firstvalue of N before repeating the routine for a reduced value of N.

After reduction of the seed size, the routine returns to step 102, andcontinues to test for token seeds. An N_(stop) value (for example, N=2)is also checked in step 110 to determine if the analysis is complete. Ifthe value of N is at N_(stop), the CPU 12 has completed a survey of theimage pixel arrays and exits the routine.

If the value of i is less than i_(max), and N is greater than N_(stop),the routine returns to step 102, and continues to test for token seeds.

When a good seed (an N×N array with approximately equal pixel values) isfound (block 104), the token is grown from the seed. In step 112, theCPU 12 pushes the pixels from the seed onto a queue. All of the pixelsin the queue are marked with the current region ID in the region map.The CPU 12 then inquires as to whether the queue is empty (decisionblock 114). If the queue is not empty, the routine proceeds to step 116.

In step 116, the CPU 12 pops the front pixel off the queue and proceedsto step 118. In step 118, the CPU 12 marks “good” neighbors around thesubject pixel, that is neighbors approximately equal in color value tothe subject pixel, with the current region ID. All of the marked goodneighbors are placed in the region map and also pushed onto the queue.The CPU 12 then returns to the decision block 114. The routine of steps114, 116, 118 is repeated until the queue is empty. At that time, all ofthe pixels forming a token in the current region will have beenidentified and marked in the region map as a Type C token. In the eventthe pixels comprise intensity histogram representations, the token canbe marked as Type C.

When the queue is empty, the CPU 12 proceeds to step 120. At step 120,the CPU 12 increments the region ID for use with identification of anext token. The CPU 12 then returns to step 106 to repeat the routine inrespect of the new current token region.

Upon arrival at N=N_(stop), step 110 of the flow chart of FIG. 5a , orcompletion of a region map that coincides with the image, the routinewill have completed the token building task. FIG. 6b is an originalimage used as an example in the identification of tokens. The imageshows areas of the color blue and the blue in shadow, and of the colorteal and the teal in shadow. FIG. 6c shows token regions correspondingto the region map, for example, as identified through execution of theroutine of FIG. 6a (Type C tokens), in respect to the image of FIG. 6b .The token regions are color coded to illustrate the token makeup of theimage of FIG. 6b , including penumbra regions between the full colorblue and teal areas of the image and the shadow of the colored areas.

Upon completion of the routine of FIG. 6a by the Type C tokenizationblock 35, the service provider 24 stores the Type C token regioninformation for the selected image. Prior to commencing any process togenerate Type B tokens from the identified Type C tokens, the operatorsblock 28 tests each identified Type C token to make certain that eachType C token encompasses a single material. While each Type C tokencomprises a region of the image having a single robust color measurementamong contiguous pixels of the image, the token may grow across materialboundaries.

Typically, different materials connect together in one Type C token viaa neck region often located on shadow boundaries or in areas withvarying illumination crossing different materials with similar hue butdifferent intensities. A neck pixel can be identified by examiningcharacteristics of adjacent pixels. When a pixel has two contiguouspixels on opposite sides that are not within the corresponding token,and two contiguous pixels on opposite sides that are within thecorresponding token, the pixel is defined as a neck pixel.

FIG. 7 shows a flow chart for a neck test for Type C tokens. In step122, the CPU 12 examines each pixel of an identified token to determinewhether any of the pixels under examination forms a neck. The routine ofFIG. 6 can be executed as a subroutine directly after a particular tokenis identified during execution of the routine of FIG. 6a . All pixelsidentified as a neck are marked as “ungrowable.” In decision block 124,the CPU 12 determines if any of the pixels were marked.

If no, the CPU 12 exits the routine of FIG. 7 and returns to the routineof FIG. 6a (step 126).

If yes, the CPU 12 proceeds to step 128 and operates to regrow the tokenfrom a seed location selected from among the unmarked pixels of thecurrent token, as per the routine of FIG. 6a , without changing thecounts for seed size and region ID. During the regrowth process, the CPU12 does not include any pixel previously marked as ungrowable. After thetoken is regrown, the previously marked pixels are unmarked so thatother tokens may grow into them.

Subsequent to the regrowth of the token without the previously markedpixels, the CPU 12 returns to step 122 to test the newly regrown token.

Neck testing identifies Type C tokens that cross material boundaries,and regrows the identified tokens to provide single material Type Ctokens suitable for use in creating Type B tokens. FIG. 6d shows Type Btokens generated from the Type C tokens of FIG. 6c , according to afeature of the present invention. The present invention provides severalexemplary techniques of pixel characteristic analysis for constructingType B tokens from Type C tokens. One exemplary technique involvesarbitrary boundary removal. The arbitrary boundary removal technique canbe applied to Type C tokens whether they were generated using N colorband values (RGB in our example) of the pixel or the filter outputrepresentation of the pixel, in the event the image was filtered. Actualboundaries of any particular Type C token will be a function of the seedlocation used to generate the token, and are thus, to some extentarbitrary. There are typically many potential seed locations for eachparticular token, with each potential seed location generating a tokenwith slightly different boundaries and spatial extent because ofdifferences among the color values of the pixels of the various seeds,within the noise ranges of the recording equipment.

FIG. 8 is a flow chart for constructing Type B tokens via an arbitraryboundary removal technique, according to a feature of the presentinvention. In step 200, the CPU 12 is provided with a set (T_(a)) ofType C tokens generated with a seed size (S) via the routine of FIG. 6a, with neck removal via the routine of FIG. 7. The seed size S=S_(max),for example, S=4 pixels. In step 202, for each Type C token, t_(c) inthe set T_(c) the CPU 12 selects a number (for example 50) of potentialseeds s₁ to s_(n). In our example, each selected seed will be a 4×4pixel array from within the token region, the pixels of the array beingof approximately equal values (within the noise levels of the recordingdevice).

In step 204, the CPU 12 grows a new Type C token, utilizing the routinesof FIGS. 6a and 7, from each seed location, s₁ to s_(n) of each tokent_(c) in the set T_(c). The newly grown tokens for each token t_(c) aredesignated as tokens r_(c1) to r_(cn). The newly grown tokens r_(c1) tor_(cn) for each token t_(c) generally overlap the original Type C tokent_(c), as well as one another.

In step 206, the CPU 12 operates to merge the newly generated tokensr_(c1) to r_(en) of each token t_(c), respectively. The result is a newtoken R_(t) corresponding to each original token t_(c) in the set T_(c).Each new token R_(t) encompasses all of the regions of the respectiveoverlapping tokens r_(c1) to r_(en) generated from the correspondingoriginal token t_(c). The unions of the regions comprising therespective merged new tokens R_(t) are each a more extensive token thanthe original Type C tokens of the set. The resulting merged new tokensR_(t) result in regions of the image file 18, each of a much broaderrange of variation between the pixels of the respective token R_(t) thanthe original Type C token, yet the range of variation among theconstituent pixels will still be relatively smooth. R_(t) is defined asa limited form of Type B token, Type B_(ab1), to indicate a tokengenerated by the first stage (steps 200-206) of the arbitrary boundaryremoval technique according to a feature of the present invention.

In step 208, the CPU 12 stores each of the Type B_(ab1) tokens generatedin steps 202-206 from the set of tokens T_(c), and proceeds to step 210.Type B_(ab1) tokens generated via execution of steps 202-206 may overlapsignificantly. In step 210, the CPU 12 operates to merge the R_(t)tokens stored in step 208 that overlap each other by a certainpercentage of their respective sizes. For example, a 30% overlap isgenerally sufficient to provide few, if any, false positive merges thatcombine regions containing different materials. The new set of mergedtokens still may have overlapping tokens, for example, previouslyoverlapping tokens that had a less than 30% overlap. After all mergesare complete, the CPU 12 proceeds to step 212.

In step 212, the CPU 12 identifies all pixels that are in more than onetoken (that is in an overlapping portion of two or more tokens). Eachidentified pixel is assigned to the token occupying the largest regionof the image. Thus, all overlapping tokens are modified to eliminate alloverlaps.

In step 214, the CPU 12 (as the Type C tokenization block 35 or theoperators block 28) stores the final set of merged and modified tokens,now designated as Type B_(ab2) tokens, and then exits the routine. Asnoted above, the Type B_(ab2) tokens were generated from Type C tokenswhether the Type C tokens were generated using N color band values (RGBin our example) of the pixel or the filter output representation of thepixel, in the event the image was filtered.

A second exemplary technique according to the present invention, forusing Type C tokens to create Type B tokens, is adjacent planar tokenmerging. The adjacent planar token merging can be implemented when animage depicts areas of uniform color, that is for non-textured regionsof an image. Initially, a token graph is used to identify tokens thatare near to one another. FIG. 9 shows a flow chart for creating a tokengraph, containing token map information, according to a feature of thepresent invention. Each token t_(c) in the set of Type C tokens T_(c),generated through execution of the routines of FIGS. 6a and 7, isevaluated in terms of a maximum distance D_(max) between tokens defininga neighboring pair of tokens, t_(c), t_(n), of the set T_(c), a minimumnumber of token perimeter pixels, P_(min), in each token of theneighboring pair of tokens, and a minimum fraction of perimeter pixels,F_(min), of each token of a neighboring pair of tokens, required to bewithin D_(max).

In step 300, the CPU 12 selects a Type C token t_(c) in the set of TypeC tokens T_(c), and identifies the pixels of the selected token t_(c)forming the perimeter of the token. In a decision block 302, the CPU 12determines whether the number of perimeter pixels is less than P_(min),for example 10 pixels.

If yes, the CPU 12 proceeds to decision block 304 to determine whetherthere are any remaining tokens t_(c) in the set of Type C tokens T_(c).If yes, the CPU 12 returns to step 300, if no, the CPU 12 exits theroutine 306.

If no, the CPU 12 proceeds to step 308. In step 308, the CPU 12generates a bounding box used as a mask to surround the selected tokent_(c) The bounding box is dimensioned to be at least D_(max) larger thanthe selected token t_(c) in all directions. A known distance transform(for example, as described in P. Felzenszwalb and D. Huttenlocher,Distance Transforms of Sampled Functions, Cornell Computing andInformation Science Technical Report TR2004-1963, September 2004), isexecuted to find the distance from each perimeter pixel of the selectedtoken t_(c) to all the pixels in the surrounding bounding box. Theoutput of the distance transform comprises two maps, each of the samesize as the bounding box, a distance map and a closest pixel map. Thedistance map includes the Euclidean distance from each pixel of thebounding box to the nearest perimeter pixel of the selected token t_(c).The closest pixel map identifies, for each pixel in the distance map,which perimeter pixel is the closest to it.

In step 310, the CPU 12 scans the distance map generated in step 308 toidentify tokens corresponding to pixels of the bounding box (from theregion map generated via the routine of FIG. 6a ), to identify a tokenfrom among all tokens represented by pixels in the bounding box, thathas a number N_(cn) of pixels within the distance D_(max), whereinN_(cn) is greater than P_(min), and greater than F_(min)*perimeterpixels of the respective token and the average distance between therespective token and t_(c) is the lowest of the tokens corresponding tothe pixels in the bounding box. If these conditions are satisfied, therespective token is designated t_(n) of a possible token pair t_(c),t_(n), and a link L_(cn) is marked active.

In step 312, the CPU 12 checks to determine whether a reciprocal linkL_(cn) is also marked active, and when it is marked active, the CPU 12marks and stores in the token graph, an indication that the token pairt_(c), t_(n) is a neighboring token pair. The reciprocal link refers tothe link status in the evaluation of the token designated as t_(n) inthe current evaluation. If that token has yet to be evaluated, the pairis not designated as a neighboring token pair until the link L_(cn) isverified as active in the subsequent evaluation of the token t_(n). TheCPU 12 then returns to decision block 304 to determine whether there areany further tokens in the set T_(c).

Upon completion of the token graph, the CPU 12 utilizes token pairinformation stored in the graph in the execution of the routine of FIG.10. FIG. 10 shows a flow chart for constructing Type B tokens via theadjacent planar token merging technique, according to a feature of thepresent invention. In the adjacent planer merging technique, pairs oftokens are examined to determine whether there is a smooth and coherentchange in color values, in a two dimensional measure, between the tokensof the pair. The color change is examined in terms of a planarrepresentation of each channel of the color, for example the RGBcomponents of the pixels according to the exemplary embodiments of thepresent invention. A smooth change is defined as the condition when aset of planes (one plane per color component) is a good fit for thepixel values of two neighboring tokens. In summary, neighboring tokensare considered the same material and a Type B token when the colorchange in a two-dimensional sense is approximately planar.

In step 320, the CPU 12 selects a token pair t_(c), t_(n) from the tokengraph. In decision block 322, the CPU 12 determines whether the meancolor in token t_(c) is significantly different from the mean color inthe token t_(c). The difference can be a function of a z-score, a knownstatistical measurement (see, for example, Abdi, H. (2007), Z-scores, inN. J. Salkind (Ed.), Encyclopedia of Measurement and Statistics,Thousand Oaks, Calif.: Sage), for example, a z-score greater than 3.0.

If the mean colors of the token pair are different, the CPU 12 proceedsto decision block 324 to determine whether there are any additionaltoken pairs in the token graph. If yes, the CPU 12 returns to step 320.If no, the CPU 12 exits the routine (step 326).

If the mean colors are within the z-score parameter, the CPU 12 proceedsto step 328. In step 328, the CPU 12 performs a mathematical operationsuch as, for example, a least median of squares regression (see, forexample, Peter J. Rousseeuw, Least Median of Squares Regression, Journalof the American Statistical Association, Vol. 79, No. 388 (December,1984), pp. 871-880) to fit a plane to each color channel of the pixels(in our example RGB) of the token pair t_(c), t_(n), as a function ofrow n and column m (see FIG. 2), the planes being defined by theequations:

R=X _(Rn) +Y _(Rm) +Z _(R)

G=X _(Gn) +Y _(Gm) +Z _(G)

B=X _(Bn) +Y _(Bm) +Z _(B)

wherein parameter values X, Y and C are determined by the least medianof squares regression operation of the CPU 12.

Upon completion of the plane fitting operation, the CPU 12 proceeds tostep 330. In step 330, the CPU 12 examines each pixel of each of thetokens of the token pair t_(c), t_(n) to calculate the z-score betweeneach pixel of the tokens and the planar fit expressed by the equation ofthe least median of squares regression operation. When at least athreshold percentage of the pixels of each token of the pair (forexample, 80%), are within a maximum z-score (for example, 0.75), thenthe neighboring token pair is marked in the token graph as indicatingthe same material in the image. After completion of step 330, the CPU 12returns to decision block 324.

Upon exiting the routine of FIG. 10, the CPU 12 examines the token graphfor all token pairs indicating the same material. The CPU 12 can achievethe examination through performance of a known technique such as, forexample, a union find algorithm. (See, for example, Zvi Galil andGiuseppe F. Italiano. Data structures and algorithms for disjoint setunion problems, ACM Computing Surveys, Volume 23, Issue 3 (September1991), pages 319-344). As a simple example, assume a set of seven Type Ctokens T₁, T₂, T₃, T₄, T₅, T₆, T₇. Assume that the result of theexecution of FIG. 9, (performance of the adjacent planar analysis),indicates that tokens T₁ and T₂ are marked as the same material, andtokens T₁ and T₃ are also marked as the same material. Moreover, theresults further indicate that tokens T₄ and T₅ are marked as the samematerial, and tokens T₅ and T₆ are also marked as the same material. Theresult of execution of the union find algorithm would therefore indicatethat tokens {T₁, T₂, T₃} form a first group within the image consistingof a single material, tokens {T₄, T₅, T₆} form a second group within theimage consisting of a single material, and token {T₇} forms a thirdgroup within the image consisting of a single material. The groups {T₁,T₂, T₃}, {T₄, T₅, T₆} and {T₇} form three Type B tokens.

A third exemplary technique according to the present invention, forusing Type C tokens to create Type B tokens, is a local token analysis.A local token approach generates Type C tokens using a window analysisof a scene depicted in an image file 18. Such tokens are designated asType C_(w) tokens. FIG. 11 is a flow chart for generating Type C_(w)tokens via the local token analysis technique, according to a feature ofthe present invention.

In step 400, the CPU 12 places a window of fixed size, for example, a33×33 pixel array mask, over a preselected series of scan positions overthe image. The window can be a shape other than a square. The scanpositions are offset from one another by a fixed amount, for example ½window size, and are arranged, in total, to fully cover the image. Thewindow area of pixels at each scan position generates a Type C_(w)token, though not every pixel within the window at the respective scanposition is in the Type C_(w) token generated at the respective scanposition.

At each scan position (step 402), the CPU 12 operates, as a function ofthe pixels within the window, to fit each of a set of planes, onecorresponding to the intensity of each color channel (for example, RGB),and an RGB line in RGB space, characterized by a start point I₀ and anend point I₁ of the colors within the window. The planar fit provides aspatial representation of the pixel intensity within the window, and theline fit provides a spectral representation of the pixels within thewindow.

For the planar fit, the planes are defined by the equations:

R=X _(Rn) +Y _(Rm) +Z _(R)

G=X _(Gn) +Y _(Gm) +Z _(G)

B=X _(Bn) +Y _(Bm) +Z _(B)

wherein parameter values X, Y and C are determined by CPU 12 byexecuting a mathematical operation such as the least median of squaresregression discussed above, a least-squares estimator, such as singularvalue decomposition, or a robust estimator such as RANSAC (see, forexample, M. A. Fischler, R. C. Bolles. Random Sample Consensus: AParadigm for Model Fitting with Applications to Image Analysis andAutomated Cartography. Comm. of the ACM, Vol 24, pp 381-395, 1981).

For the RGB line fit, the line is defined by:I(r,g,b)=I₀(r,g,b)+t(I₁(r,g,b)−I₀(r,g,b)) wherein the parameter t has avalue between 0 and 1, and can be determined by the CPU 12 utilizing anyof the mathematical techniques used to find the planar fit.

At each scan position, after completion of step 402, the CPU 12 operatesin step 404 to examine each pixel in the window in respect of each ofthe planar fit representation and RGB line representation correspondingto the respective window scan position. For each pixel, the CPU 12determines an error factor for the pixel relative to each of theestablished planes and RGB line. The error factor is related to theabsolute distance of the pixel to its projection on either from eitherthe planar fit or the RGB line fit. The error factor can be a functionof the noise present in the recording equipment or be a percentage ofthe maximum RGB value within the window, for example 1%. Any pixeldistance within the error factor relative to either the spatial planarfit or the spectral line fit is labeled an inlier for the Type C_(w)token being generated at the respective scan position. The CPU 12 alsorecords for the Type C_(w) token being generated at the respective scanposition, a list of all inlier pixels.

At each scan position, after completion of step 404, the CPU 12 operatesin step 406 to assign a membership value to each inlier pixel in thewindow. The membership value can be based upon the distance of theinlier pixel from either the planar fit or the RGB line fit. In oneexemplary embodiment of the present invention, the membership value isthe inverse of the distance used to determine inlier status for thepixel. In a second exemplary embodiment, a zero-centered Gaussiandistribution with a standard deviation is executed to calculatemembership values for the inlier pixels.

After all of the scan positions are processed to generate the Type C_(w)tokens, one per scan position, the CPU 12 operates to compile and storea token data list (step 408). The token data list contains two lists. Afirst list lists all of the pixels in the image file 18, and for eachpixel, an indication of each Type C_(w) token to which it labeled as aninlier pixel, and the corresponding membership value. A second listlists all of the generated Type C_(w) tokens, and for each token anindication of the inlier pixels of the respective token, and thecorresponding membership value. After compiling and storing the tokendata list, the CPU 12 exits the routine (step 410).

FIG. 12 is a flow chart for constructing Type B tokens from the TypeC_(w) tokens generated via the local token analysis technique, accordingto a feature of the present invention. In step 420, the CPU 12calculates a similarity of parameters of the spatial planer dimensionsand spectral RGB lines of adjacent or overlapping Type C_(w) tokensgenerated through execution of the routine of FIG. 108. Overlapping andadjacent Type C_(w) tokens can be defined as tokens corresponding toscan positions that overlap or are contiguous. A similarity thresholdcan be set as a percentage of difference between each of the spatialplaner dimensions and spectral RGB lines of two overlapping or adjacentType C_(w) tokens being compared. The percentage can be a function ofthe noise of, for example, the camera 14 used to record the scene of theimage file 18. All overlapping or adjacent Type C_(w) token pairs havinga calculated similarity within the similarity threshold are placed on alist.

In step 422, the CPU 12 sorts the list of overlapping or adjacent TypeC_(w) token pairs having a calculated similarity within the similaritythreshold, in the order of most similar to least similar pairs. In step424, the CPU 12 merges similar token pairs, in the order of the sort,and labeling pairs as per degree of similarity. Each merged token pairwill be considered a Type_(B) token. In step 426, the CPU 12 stores thelist of Type_(B) tokens, and exits the routine.

A fourth exemplary technique according to the present invention, forcreating Type B tokens, is a technique using log chromaticityclustering. for constructing Type B tokens for an image file 18. Logchromaticity is a technique for developing an illumination invariantchromaticity space. As noted above, a method and system for separatingillumination and reflectance using a log chromaticity representation isdisclosed in U.S. Pat. No. 7,596,266. The techniques taught in U.S. Pat.No. 7,596,266 can be used to provide illumination invariant logchromaticity representation values for each color of an image, forexample, as represented by Type C tokens. Logarithmic values of thecolor band values of the image pixels are plotted on a log-color spacegraph. The logarithmic values are then projected to a log-chromaticityprojection plane oriented as a function of the bi-illuminant dichromaticreflection model (a BIDR normal), to provide a log chromaticity valuefor each pixel.

FIG. 13 is a graphic representation of a log color space, bi-illuminantchromaticity plane according to a feature of the invention disclosed inU.S. Pat. No. 7,596,266. The alignment of the chromaticity plane isdetermined by a vector N, normal to the chromaticity plane, and definedas N=log(Bright_(vector))−log(Dark_(vector))=log(1+1/S_(vector)). Theco-ordinates of the plane, u, v can be defined by a projection of thegreen axis onto the chromaticity plane as the u axis, and the crossproduct of u and N being defined as the v axis. In our example, each logvalue for the materials A, B, C is projected onto the chromaticityplane, and will therefore have a corresponding u, v co-ordinate value inthe plane that is a chromaticity value, as shown in FIG. 5.

Thus, according to the technique disclosed in U.S. Pat. No. 7,596,266,the RGB values of each pixel in an image file 18 can be mapped by theCPU 12 from the image file value p(n, m, R, G, B) to a log value, then,through a projection to the chromaticity plane, to the corresponding u,v value, as shown in FIG. 13. Each pixel p(n, m, R, G, B) in the imagefile 18 is then replaced by the CPU 12 by a two dimensional chromaticityvalue: p(n, m, u, v), to provide a chromaticity representation of theoriginal RGB image. In general, for an N band image, the N color valuesare replaced by N−1 chromaticity values. The chromaticity representationis a truly accurate illumination invariant representation because theBIDR model upon which the representation is based, accurately andcorrectly represents the illumination flux that caused the originalimage.

According to a feature of the present invention, log chromaticity valuesare calculated for each color depicted in an image file 18 input to theCPU 12 for identification of regions of the uniform reflectance (Type Btokens). For example, each pixel of a Type C token will be ofapproximately the same color value, for example, in terms of RGB values,as all the other constituent pixels of the same Type C token, within thenoise level of the equipment used to record the image. Thus, an averageof the color values for the constituent pixels of each particular Type Ctoken can be used to represent the color value for the respective Type Ctoken in the log chromaticity analysis.

FIG. 14 is a flow chart for determining a list of colors depicted in aninput image, for example, an image file 18. In step 500, an input imagefile 18 is input to the CPU 12 for processing. In steps 502 and 504, theCPU 12 determines the colors depicted in the input image file 18. Instep 502, the CPU 12 calculates an average color for each Type C tokendetermined by the CPU 12 through execution of the routine of FIG. 6a ,as described above, for a list of colors. The CPU 12 can be operated tooptionally require a minimum token size, in terms of the number ofconstituent pixels of the token, or a minimum seed size (the N×N array)used to determine Type C tokens according to the routine of FIG. 6a ,for the analysis. The minimum size requirements are implemented toassure that color measurements in the list of colors for the image arean accurate depiction of color in a scene depicted in the input image,and not an artifact of blend pixels.

Blend pixels are pixels between two differently colored regions of animage, as noted above. If the colors between the two regions are plottedin RGB space, there is a linear transition between the colors, with eachblend pixel, moving from one region to the next, being a weightedaverage of the colors of the two regions. Thus, each blend pixel doesnot represent a true color of the image. The CPU 12 can eliminate tokensconsisting of blend pixel from the analysis via the blend pixel imagecreated in step 1018 of FIG. 3 c.

In step 504, the CPU 12 can alternatively collect colors at the pixellevel, that is, the RGB values of the pixels of the input image file 18,as shown in FIG. 2. The CPU 12 can be operated to optionally requireeach pixel of the image file 18 used in the analysis to have a minimumstability or local standard deviation via a filter output, for a moreaccurate list of colors. For example, second derivative energy can beused to indicate the stability of pixels of an image.

In this approach, the CPU 12 calculates a second derivative at eachpixel, or a subset of pixels disbursed across the image to cover allillumination conditions of the image depicted in an input image file 18,using a Difference of Gaussians, Laplacian of Gaussian, or similarfilter. The second derivative energy for each pixel examined can then becalculated by the CPU 12 as the average of the absolute value of thesecond derivative in each color band (or the absolute value of thesingle value in a grayscale image), the sum of squares of the values ofthe second derivatives in each color band (or the square of the singlevalue in a grayscale image), the maximum squared second derivative valueacross the color bands (or the square of the single value in a grayscaleimage), or any similar method. Upon the calculation of the secondderivative energy for each of the pixels, the CPU 12 analyzes the energyvalues of the pixels. There is an inverse relationship between secondderivative energy and pixel stability, the higher the energy, the lessstable the corresponding pixel.

In step 506, the CPU 12 outputs a list or lists of color (afterexecuting one or both of steps 502 and/or 504). According to a featureof the present invention, all of the further processing can be executedusing the list from either step 502 or 504, or vary the list used (oneor the other of the lists from steps 502 or 504) at each subsequentstep.

FIG. 15 is a flow chart for determining an orientation for a logchromaticity representation, according to a feature of the presentinvention. For example, the CPU 12 determines an orientation for thenormal N, for a log chromaticity plane, as shown in FIG. 5. In step 210,the CPU 12 receives a list of colors for an input file 18, such as alist output in step 506 of the routine of FIG. 14. In step 512, the CPU12 determines an orientation for a log chromaticity space.

As taught in U.S. Pat. No. 7,596,266, and as noted above, alignment ofthe chromaticity plane is represented by N, N being a vector normal tothe chromaticity representation, for example, the chromaticity plane ofFIG. 13. The orientation is estimated by the CPU 12 thorough executionof any one of several techniques. For example, the CPU 12 can determineestimates based upon entropy minimization, manual selection by a user orthe use of a characteristic spectral ratio for an image of an inputimage file 18, as fully disclosed in U.S. Pat. No. 7,596,266.

For a higher dimensional set of colors, for example, an RYGB space (red,yellow, green, blue), the log chromaticity normal, N, defines asub-space with one less dimension than the input space. Thus, in thefour dimensional RYGB space, the normal N defines a three dimensionallog chromaticity space. When the four dimensional RYGB values areprojected into the three dimensional log chromaticity space, theprojected values within the log chromaticity space are unaffected byillumination variation.

In step 514, the CPU 12 outputs an orientation for the normal N. Asillustrated in the example of FIG. 13, the normal N defines anorientation for a u, v plane in a three dimensional RGB space.

FIG. 16 is a flow chart for determining log chromaticity coordinates forthe colors of an input image, as identified in steps 502 or 504 of theroutine of FIG. 14, according to a feature of the present invention. Instep 520, a list of colors is input to the CPU 12. The list of colorscan comprise either the list generated through execution of step 502 ofthe routine of FIG. 14, or the list generated through execution of step504. In step 522, the log chromaticity orientation for the normal, N,determined through execution of the routine of FIG. 15, is also input tothe CPU 12.

In step 524, the CPU 12 operates to calculate a log value for each colorin the list of colors and plots the log values in a three dimensionallog space at respective (log R, log G, log B) coordinates, asillustrated in FIG. 13. Materials A, B and C denote log values forspecific colors from the list of colors input to the CPU 12 in step 520.A log chromaticity plane is also calculated by the CPU 12, in the threedimensional log space, with u, v coordinates and an orientation set byN, input to the CPU 12 in step 522. Each u, v coordinate in the logchromaticity plane can also be designated by a corresponding (log R, logG, log B) coordinate in the three dimensional log space.

According to a feature of the present invention, the CPU 12 thenprojects the log values for the colors A, B and C onto the logchromaticity plane to determine a u, v log chromaticity coordinate foreach color. Each u, v log chromaticity coordinate can be expressed bythe corresponding (log R, log G, log B) coordinate in the threedimensional log space. The CPU 12 outputs a list of the log chromaticitycoordinates in step 526. The list cross-references each color to a u, vlog chromaticity coordinate and to the pixels (or a Type C tokens)having the respective color (depending upon the list of colors used inthe analysis (either step 502 (tokens) or 504 (pixels))).

FIG. 17 is a flow chart for optionally augmenting the log chromaticitycoordinates for pixels or Type C tokens with extra dimensions, accordingto a feature of the present invention. In step 530, the list of logchromaticity coordinates, determined for the colors of the input imagethrough execution of the routine of FIG. 16, is input to the CPU 12. Instep 532, the CPU 12 accesses the input image file 18, for use in theaugmentation.

In step 534, the CPU 12 optionally operates to augment each logchromaticity coordinate with a tone mapping intensity for eachcorresponding pixel (or Type C token). The tone mapping intensity isdetermined using any known tone mapping technique. An augmentation withtone mapping intensity information provides a basis for clusteringpixels or tokens that are grouped according to both similar logchromaticity coordinates and similar tone mapping intensities. Thisimproves the accuracy of a clustering step.

In step 536, the CPU 12 optionally operates to augment each logchromaticity coordinate with x, y coordinates for the correspondingpixel (or an average of the x, y coordinates for the constituent pixelsof a Type C token) (see FIG. 2 showing a P (1,1) to P (N, M) pixelarrangement). Thus, a clustering step with x, y coordinate informationwill provide groups in a spatially limited arrangement, when thatcharacteristic is desired.

In each of steps 534 and 536, the augmented information can, in eachcase, be weighted by a factor w₁ and w₂, w₃ respectively, to specify therelative importance and scale of the different dimensions in theaugmented coordinates. The weight factors w, and w₂, w₃ areuser-specified. Accordingly, the (log R, log G, log B) coordinates for apixel or Type C token is augmented to (log R, log G, log B, T*w₁, x*w₂,y*w₃) where T, x and y are the tone mapped intensity, the x coordinateand the y coordinate, respectively.

In step 538, the CPU 12 outputs a list of the augmented coordinates. Theaugmented log chromaticity coordinates provide accurate illuminationinvariant representations of the pixels, or for a specified regionalarrangement of an input image, such as, for example, Type C tokens.According to a feature of the present invention, the illuminationinvariant characteristic of the log chromaticity coordinates is reliedupon as a basis to identify regions of an image of a single material orreflectance, such as, for example, Type B tokens.

FIG. 18 is a flow chart for clustering the log chromaticity coordinates,according to a feature of the present invention. In step 540, the listof augmented log chromaticity coordinates is input the CPU 12. In step542, the CPU 12 operates to cluster the log chromaticity coordinates.The clustering step can be implemented via, for example, a known k-meansclustering. Any known clustering technique can be used to cluster thelog chromaticity coordinates to determine groups of similar logchromaticity coordinate values. The CPU 12 correlates each logchromaticity coordinate to the group to which the respective coordinatebelongs. The CPU 12 also operates to calculate a center for each groupidentified in the clustering step. For example, the CPU 12 can determinea center for each group relative to a (log R, log G, log B, log T)space.

In step 544, the CPU 12 outputs a list of the cluster group membershipsfor the log chromaticity coordinates (cross referenced to either thecorresponding pixels or Type C tokens) and/or a list of cluster groupcenters.

As noted above, in the execution of the clustering method, the CPU 12can use the list of colors from either the list generated throughexecution of step 502 of the routine of FIG. 14, or the list generatedthrough execution of step 504. In applying the identified cluster groupsto an input image, the CPU 12 can be operated to use the same set ofcolors as used in the clustering method (one of the list of colorscorresponding to step 502 or to the list of colors corresponding to step504), or apply a different set of colors (the other of the list ofcolors corresponding to step 502 or the list of colors corresponding tostep 504). If a different set of colors is used, the CPU 12 proceeds toexecute the routine of FIG. 19.

FIG. 19 is a flow chart for assigning the log chromaticity coordinatesto clusters determined through execution of the routine of FIG. 18, whena different list of colors is used after the identification of thecluster groups, according to a feature of the present invention. In step550, the CPU 12 once again executes the routine of FIG. 8, this time inrespect to the new list of colors. For example, if the list of colorsgenerated in step 502 (colors based upon Type C tokens) was used toidentify the cluster groups, and the CPU 12 then operates to classifylog chromaticity coordinates relative to cluster groups based upon thelist of colors generated in step 504 (colors based upon pixels), step550 of the routine of FIG. 19 is executed to determine the logchromaticity coordinates for the colors of the pixels in the input imagefile 18.

In step 552, the list of cluster centers is input to the CPU 12. In step554, the CPU 12 operates to classify each of the log chromaticitycoordinates identified in step 550, according to the nearest clustergroup center. In step 556, the CPU 12 outputs a list of the clustergroup memberships for the log chromaticity coordinates based upon thenew list of colors, with a cross reference to either correspondingpixels or Type C tokens, depending upon the list of colors used in step550 (the list of colors generated in step 502 or the list of colorsgenerated in step 504).

FIG. 20 is a flow chart for detecting regions of uniform reflectancebased on the log chromaticity clustering according to a feature of thepresent invention. In step 560, the input image file 18 is once againprovided to the CPU 12. In step 562, one of the pixels or Type C tokens,depending upon the list of colors used in step 550, is input to the CPU12. In step 564, the cluster membership information, from either steps544 or 556, is input to the CPU 12.

In step 566, the CPU 12 operates to merge each of the pixels, orspecified regions of an input image, such as, for example, Type Ctokens, having a same cluster group membership into a single region ofthe image to represent a region of uniform reflectance (Type B token).The CPU 12 performs such a merge operation for all of the pixels ortokens, as the case may be, for the input image file 18. In step 568,the CPU 12 outputs a list of all regions of uniform reflectance (andalso of similar tone mapping intensities and x, y coordinates, if thelog chromaticity coordinates were augmented in steps 234 and/or 236). Itshould be noted that each region of uniform reflectance (Type B token)determined according to the features of the present invention,potentially has significant illumination variation across the region.

In a further exemplary embodiment of the present invention, the CPU 12(executing as the operators block 28) compiles lists of Type B tokensseparately generated through each of and/or a combination of one or moreof the arbitrary boundary removal, adjacent planar token merging, localtoken analysis and log chromaticity clustering techniques. Thedetermination of the combination of techniques used depends in part onwhether a particular region of the image was filtered because oftexturing of the image. Since each Type B token generated through thedescribed techniques likely represents a single material under varyingillumination conditions, merging sufficiently overlapping Type B tokensgenerated through the use of varying and different techniques, providesa resulting, merged Type B token that represents a more extensive areaof the image comprising a single material, and approaches the extent ofa Type A token.

Sufficiently overlapping can be defined by satisfaction of certain pixelcharacteristic criteria, such as, for example:

A) The two Type B tokens have at least n of the original Type C tokensin common, for example, n=1B) The two Type B tokens have at least n pixels in common, for example,n=20C) The two Type B tokens have at least n % overlap, that is at least n %of the pixels in a first one of the two Type B tokens are also found inthe second one of the two Type B tokens or vice versa, wherein, forexample n %=10%.D) The percentage of pixels in a smaller one of the two Type B tokens,also found in the larger one of the two Type B tokens is above apreselected threshold, for example 15%.E) A preselected combination of criteria A-D.

Merging of two sufficiently overlapping Type B tokens can beaccomplished via a mathematical operation such as execution of the unionfind algorithm discussed above. In the case of two overlapping Type Btokens that do not satisfy the above discussed criteria, the overlappingpixels of the two tokens can be assigned to the larger one of the twoType B tokens.

Referring once again to FIG. 4, as a result of execution by the Type Ctokenization block 35 and/or the operators block 28 (via the CPU 12) ofthe token generation and merging techniques according to features of thepresent invention, an image can be accurately segmented into tokensrepresenting discrete materials depicted in the scene (Type B tokens)and tokens representing regions of robust similar color (Type C tokens),thus providing a basis for computational efficiencies, as the tokenrepresentations capture spatio-spectral information of a significantnumber of constituent pixels. The service provider 24 stores all of theType C and Type B tokens generated through execution of the abovedescribed token generation techniques, along with the relevant token mapinformation, for example, as determined during execution of the adjacentplanar token merging technique, and cross-references the stored operatorresults to the associated selected image file 18, for use in anysegregation processing of the selected image. The cross referenceinformation also includes information on the various selectively variedrepresentations (image files 18 a . . . 18 n) that may have been subjectto a tokenization operation.

In our example of a same illumination constraint, the service provider24 identifies Type C and Type B tokens as the operators required by theselected constraint. The Type C tokenization block 35 generated the TypeC tokens. The service provider 24 operates the operators block 28 toexecute the above described techniques, to generate the relevant Type Btokens for the image 32, as well as a token map. The constraint builder26 organizes the generated token operators according to the exemplarymatrix equation, [A] [x]=[b], for input to the solver 30. In the sameillumination constraint, the constraining relationship of the relevantconstraint generator software module is that adjacent Type C tokens, asindicated by the token map information, are lit by the sameillumination, unless the adjacent Type C tokens are part of the sameType B token.

According to the multi-resolution exemplary embodiment of the presentinvention, the tokenization processes can be executed by the CPU 12 inrespect to the original image file 18 or any of the selectively variedrepresentations, image files 18 a . . . 18 n, and constraints applied atany level.

Each Type C token stored by the service provider 24 is identified by aregion ID, and includes a listing of each constituent pixel by row andcolumn number (and a cross reference to the associated image filewherein the token was identified (image files 18, 18 a . . . 18 n)).Each pixel of a Type C token will be of approximately the same colorvalue, for example, in terms of RGB values, as all the other constituentpixels of the same Type C token, within the noise level of the equipmentused to record the image. An average of the color values for theconstituent pixels of each particular Type C token can be used torepresent the color value for the respective Type C token. Each Type Btoken is identified by constituent Type C tokens, and thus can beprocessed to identify all of its constituent pixels via the respectiveconstituent Type C tokens.

Pursuant to a feature of the present invention, a model for imageformation reflects the basic concept of an image as comprising twocomponents, material and illumination. This relationship can beexpressed as: I=ML, where I is the image color, as recorded and storedin the respective image file 18 and/or convolved in representations 18 a. . . 18 n, M the material component of the recorded image color and Lthe illumination component of the recorded image color. The I value foreach Type C token is therefore the average color value for the recordedcolor values of the constituent pixels of the token.

Thus: log(I)=log (ML)=log (M)+log(L). This can be restated as i=m+l,where i represents log(I), m represents log(M) and l represents log(L).In the constraining relationship of the same illumination constraint, inan example where three Type C tokens, a, b and c, (see FIG. 21) areadjacent (and not within the same Type B token, (as can be shown by acomparison of row and column numbers for all constituent pixels)),I_(a)=I_(b)=I_(c). Since: I_(a)=I_(a)−m_(a), I_(b)=I_(b)−m_(b), andI_(c)=I_(c)−M_(c), these mathematical relationships can be expressed as(1)m_(a)+(−1)m_(b)+(0)m_(c)=(i_(a)−i_(b)),(1)m_(a)+(0)m_(b)+(−1)m_(c)=(i_(a)−i_(c)) and(0)m_(a)+(1)m_(b)+(−1)m_(c)=(i_(b)−i_(c)).

FIG. 21 shows a representation of an [A] [x]=[b] matrix equation for themathematical relationships of the example of the three adjacent Type Ctokens a, b and c described above, as constrained by the sameillumination constraint: the adjacent Type C tokens a, b and c are atthe same illumination. In the matrix equation of FIG. 15, the variousvalues for the log (I), in the [b] matrix, are known from the averagerecorded pixel color values for the constituent pixels of the adjacentType C tokens a, b and c, generated by the Type C tokenization block 35from the image selected for segregation. The [A] matrix of 0's, 1's and−1's, is defined by the set of equations expressing the selected sameillumination constraint, as described above. The number of rows in the[A] matrix, from top to bottom, corresponds to the number of actualconstraints imposed on the tokens, in this case three, the sameillumination between three adjacent Type C tokens. The number of columnsin the [A] matrix, from left to right, corresponds to the number ofunknowns to be solved for, again, in this case, three. Therefore, thevalues for the material components of each Type C token a, b and c, inthe [x] matrix, can be solved for in the matrix equation. It should benoted that each value is actually a vector of three values correspondingto the RGB color bands of our example.

Accordingly, the matrix equation of FIG. 21, as arranged by theconstraint builder 26, is input by the constraint builder 26 to thesolver 30 for an optimized solution for the values of the materialcomponents of the adjacent Type C tokens a, b and c of the selectedimage. As noted above, in the exemplary GUI embodiment of the presentinvention, a user selects one of several mathematical techniques forfinding the optimal solution to the system of constraint equations, [A][x]=[b]. The CPU 12 configures the solver 30 according to themathematical operation selected by the user.

For example, in a standard least squares solver, the matrix equation isrestated as \underset{x}{min}(Ax−b)². The solver 30 then executes theleast squares operation to determine optimized values for each of m_(a),m_(b) and m_(c). The solver 30 can then proceed to generate and displaya material image based upon the optimal m_(a), m_(b) and m_(c) values.In the material image, the m_(a), m_(b) and m_(c) values are substitutedfor the originally recorded RGB values, for each pixel of the respectivetokens. The solver 30 can proceed to also generate an illumination imagefrom the known recorded image values i_(a), i_(b), i_(c), and thedetermined m_(a), m_(b) and m_(c) values, utilizing the model expressedby i=m+1.

Each of the material and illumination images are displayed on themonitor 20, via, for example, the GUI (see FIG. 5) and can be stored bythe service provider 24, and cross-referenced to the original image file18. According to a feature of the present invention, intrinsic imagesgenerated from the representations image files 18 a . . . 18 n byperformance of the above described spatio-spectraloperator/constraint/solver process, are also stored by the serviceprovider 24.

As noted above, the spatio-spectral operator/constraint/solver processcan be efficiently executed by the CPU 12 first in respect of therelatively coarse representation depicted in image file 18 a.Thereafter, the solver (executing via the CPU 12) is constrained by theweighted average for color equation described above, when solving for anillumination image and/or material image for the fine scale originalimage file 18. Moreover, according to a further exemplary embodiment ofthe present invention, the spatio-spectral operator/constraint/solverprocess can be executed by the CPU 12 selectively relative to thevarious representations of the image. For example, only regions of theimage file 18 having edges, are tokenized, while all regions of uniformcolor within the relatively coarse image file 18 a are tokenized, for adivision of processing, as described above.

FIG. 22 shows a representation of an [A] [x]=[b] matrix equation forthree adjacent Type C tokens a, b and c, as constrained according to afurther exemplary embodiment of the present invention. The sameillumination constraint discussed above, with reference to FIG. 21,enforces a strict solution between the tokens a, b and c, according tothe same illumination constraint imposed upon the tokens in theillustrated example. In the further exemplary embodiment of the presentinvention, a soft, weighted constraint can be used to provide a moreaccurate segregation of an image into the intrinsic material reflectanceand illumination components. The constraint of the further exemplaryembodiment is based upon the BIDR model and constrains all color bandvariations into one integral constraining relationship.

According to the further exemplary embodiment of the present invention,each constraint between any two image locations, for example, Type Ctokens i and j, is expressed by the log color space equation:w_(ij)*(Mi−Mj)=w_(ij)*(a_(ij)V_(ij)+(1−α_(ij))B_(ij)), where Mi and Mjare log color space values for the material reflectance component of thetokens i and j, respectively, V_(ij) is a difference vector to provide agradient magnitude defined by V_(ij)=I_(i)−I_(j), where I_(i) and I_(j)are the recorded log color space values for the color and intensity ofthe tokens i and j, respectively, B_(ij) is the component of V_(ij) thatis not aligned with illumination change and is a vector defined byB_(ij)=V_(ij)−(V_(ij)·N), where N is the BIDR normal, as describedabove, α_(ij) is a scalar value based upon a smooth scalar function αthat varies from 0 to 1, as a function of the degree that the changeexpressed by I_(i)−I_(j) corresponds to an illumination change or amaterial reflectance change, and w_(ij) is a constraint confidenceweight that specifies the importance of the constraint in the solve ofthe [A] [x]=[b] matrix equation.

Thus, the w_(ij)*(Mi−Mj)=w_(ij)*(α_(ij)V+(1−α_(ij))B_(ij)) constraintequation, according to the present invention, provides a flexible,weighted constraint relationship that accurately reflects the nature ofcolor change, as a function of spatio-spectral information representedby the BIDR model, between any image locations, for example, between thelocations of tokens i and j, and expresses color change as caused by anillumination change, or a material reflectance change or a combinationof both. The constraint relationship therefore can be implemented in asolve that results in a more precise and accurate segregation of animage into the intrinsic material reflectance and illuminationcomponents. The image locations selected for inclusion in the constraintequation can be adjacent locations, or locations separated by a fixedamount, for example, 2, 3 or 4 pixels apart. A group of locations canalso be selected, for example, all tokens that are 2 to 10 pixels apartfrom the ith token.

For example, assuming a constraint confidence weight of 1, then whenα_(ij)=1, the equation reduces to a same illumination constraint,Mi−Mj=V_(ij). In this form, the difference between the materialreflectance components of the two tokens i and j equals all of the colorchange expressed by the vector V_(ij), due to the same illuminationacross the two tokens.

Again assuming a constraint confidence weight of 1, when α_(ij)=0, theequation reduces to a same material constraint, Mi−Mj=B_(ij). In thisform, the difference between the two tokens is almost the same, exceptfor the value of B_(ij), the component of color change that is notaligned with the BIDR normal, and therefore expresses some difference inmaterial reflectance.

In addition, the equation can be expressed in terms of an anchorconstraint, when material reflectance difference, or the materialreflectance itself, is constrained to a pre-determined value x. In thatcase, the equation is expressed as Mi−Mj=x, when the difference inmaterial reflectance between two image locations is anchored to apre-determined amount, and Mi=x, when a single location, for example, atoken, is constrained to a pre-determined amount.

The w_(ij)*(Mi−Mj)=w_(ij)*(α_(ij)V_(ij)+(1−α_(ij))B_(ij)) constraintequation, according to the present invention, relies upon purity ofcolor values for an accurate measure of color change due toillumination. In that connection, the known Gaussian and Laplaciantechniques for creating the pyramids, as described above, tend toaverage or blur colors from a high resolution to the next, lowerresolution level. A pyramid arrangement that preserves the purity ofcolor from scale to scale is needed to insure the accuracy of solveresults.

Thus, according to a feature of the present invention, to increaseaccuracy in a solve based upon constraints, a simple mean method tocreate the pyramid is used to assure color purity at each of and betweenthe pyramid levels.

Referring now to FIG. 27, there is shown a flow chart for a simple meanmethod for color pure down sampling according to a feature of thepresent invention. The routine of FIG. 27 provides a downsamplingcorrespondence map, as illustrated in FIG. 28. FIG. 28 is a graphicrepresentation of pixel locations at different scales of a scale-spacedpyramid, as shown in FIG. 3b . The area designated as pixel i_(k+1) inthe lower resolution level scale k+1, when downsampling to the higherresolution level k, will be assigned the color of the pixel i_(k) of thecorresponding 2×2 pixel region of the higher resolution level k.

According to a feature of the present invention, the selection of thepixel i_(k) in the corresponding 2×2 pixel region, for color accuratedownsampling mapping, is executed by the CPU 12 via the routine of FIG.27. The inputs for the CPU 12 to execute the routine of FIG. 27 includethe original image file 18, and the downsampling scale D (1600). In theexample of FIG. 28, the downsampling scale is 2, as a 2×2 pixelneighborhood of the high resolution level k is represented by a singlelocation, pixel i_(k)+1 of the k+1 level.

In step 1602, the CPU 12 operates to select the D×D neighborhoodcorresponding to an image location for a low resolution, higher scale,for example, as shown in FIG. 28. In step 1604, the CPU 12 calculatesthe mean color for the pixels in the D×D neighborhood. In step 1606, theCPU 12 operates to select the pixel in the D×D neighborhood that mostclosely matches the mean color value calculated in step 1604. In theexample of FIG. 28, the CPU 12 selects pixel i_(k). In step 1608, theCPU 12 maps the color of the selected pixel from the D×D neighborhood ofthe high resolution level to the current location of the low levelresolution, links pixels i_(k) and i_(k+1), and then stores the mapindication (step 1610).

In step 1612, the CPU 12 repeats steps 1602-1610, for all of the otherlocations of the current level of the pyramid. When the map for thecurrent level is complete, the CPU 12 proceeds to decision block 1614.In decision block 1614, the CPU 12 checks the current level to determineif it is less than the resolution of the lowest resolution level of thepyramid. If no, the CPU proceeds back to step 1602, and begins theroutine again, up the pyramid, at the next, lower resolution level ofthe pyramid. If yes, the CPU 12 proceeds to step 1616, and exits theroutine.

Referring now back to FIG. 3c , step 1038 b is executed by the CPU 12 byselecting locations, for example three tokens a, b, c, at a scale, forexample, depicted at one of the levels of the token pyramid, createdaccording to the simple mean method, input from block 1016 (as shown inFIG. 3b ). The CPU 12 also accesses the corresponding locations of theblend pixel pyramid, input from block 1020, to verify that the selectedtokens are not blend pixel tokens. In the event that the selected tokensare blend pixel tokens, the CPU 12 can operate to take an actionrelative to the blend pixel tokens. For example, the CPU 12 caneliminate the selected tokens from any constraint, or limit theinclusion to certain types of constraints, for example, only samematerial constraints. Moreover, the CPU 12 accesses the normal imagepyramid, input from block 1026, to obtain BIDR normal information forthe selected tokens a, b, c.

As shown in the example of FIG. 22, the CPU 12 completes step 1038 b byorganizing the constraints relative to the tokens a, b, c in the [A][x]=[b] matrix equation, according to thew_(ij)*(Mi−Mj)=w_(ij)*(α_(ij)V_(ij)+(1−α_(ij))B_(ij)) weightedconstraint equation. The CPU 12 also calculates the values for w_(ij)and α_(ij) as well as the values for V_(ij) and B_(ij).

Referring now to FIG. 23, there is shown a flow chart for executing theweighted constraint equation as a BIDR alignment constraint according toa feature of the present invention. The flow chart depicts operation bythe CPU 12 for calculating the values for w_(ij) and α_(ij) and V_(ij)and B_(ij), for example, to complete step 1038 b, in spatial constraintsbetween locations of one level of the image pyramid, as shown in FIG.25a . In step 1500, the CPU 12 receives as input the average recordedcolor value (I_(i) and I_(j)) for the constituent pixels of each oftokens i and j, where i and j are one of the ab, ac, or be pairs fromthe a, b, c tokens of the example of FIG. 22.

In step 1500, the CPU 12 operates to compute the difference vector V byexecuting the equation V_(ij)=I_(i)−I_(j), as described above. Theresult of step 1500 is input to block 1502. The BIDR normal N accessedby the CPU 12 from corresponding locations of the normal pyramid is alsoinput to block 1502. In block 1502, the CPU 12 applies the inputs (V andN) to compute the vector B_(ij) by executing the equationB_(ij)=V_(ij)−(V_(ij)·N). The result of step 1500 is also input todecision block 1504.

In decision block 1504, a user sets a selection to one of two differentmodes corresponding to two different metrics for calculating w_(ij) andα_(ij), an angle metric and a distance metric. In the event the anglemetric is selected by a user, the CPU 12 proceeds to step 1506.

In step 1506, the CPU 12 computes the angle metric, a BIDR alignmentangle θ, the angle between the vector V and the BIDR normal N, θ=cos⁻¹(V·N).

In the event the distance metric is selected by a user, the CPU 12proceeds to step 1508. In step 1508, the CPU 12 computes the distancemetric d, measured as the distance between the projections on the logchromaticity plane (the u, v plane perpendicular to the BIDR normal, asshown in FIG. 13) of the log color space values I_(i) and I_(j).

In step 1510, the CPU 12 receives either the angle metric θ (step 1506)or the distance metric d (step 1506), depending on the user selection(step 1504). In step 1510, the CPU 12 applies a sigmoid function toobtain values for w_(ij) and α_(ij).

To that end, a user selects a sigmoid slope s (1512) and a sigmoidthreshold τ (1514). These parameters set the rate and the point oftransition of the sigmoid function. The sigmoid slope s is input to step1510. In one exemplary embodiment of the present invention, the sigmoidthreshold τ, set by the user, can be scaled, based upon the magnitude ofthe vector V (block 1516).

In the scaled threshold embodiment (block 1516), the sigmoid threshold τset by the user is input to a decision block 1518. A user sets anindication whether to use illumination smoothness in the constraintsconcatenated in the [A] [x]=[b] matrix. If no, the CPU 12 proceeds tostep 1524, for input to step 1510, and the threshold τ is applied as afixed value, without modification.

If yes, the sigmoid threshold τ is input to step 1520. In step 1520, theCPU 12 references step 1504 to determine the mode selected by a user,the angle metric mode or the distance metric mode. Depending on theselected mode, the CPU 12 then references a threshold scale graph, forexample, as shown in FIGS. 26a and b . The threshold graphs shown inFIGS. 26a and b can be adjusted according to an illumination smoothnessvalue input from block 1522. FIG. 26a shows the scale adjustment for thethreshold τ for different values of the vector V when the angle metricis used and FIG. 26b shows the scale adjustment for the threshold τ fordifferent values of the vector V when the distance metric is used. TheCPU 12 operates to output the appropriate threshold value to step 1510.

In step 1510, the CPU 12 applies a sigmoid function, as noted above, ineither the angle metric mode or distance metric mode, using the valuesfor s and t input from steps 1512 and 1516, respectively, and either theangle or distance value from one or the other of steps 1506 and 1508.

In the angle metric mode, the CPU 12 executes a sigmoid function tocompute α_(ij) as a function of θ: α_(ij)=1/1+e^(−s*(θ−τ)). FIG. 24shows a plot of the sigmoid function for the angle metric mode, forvalues of 0 from 00 to 450. As shown in FIG. 24, for low values of θ,for example 10° and lower, the value of α_(ij) is 0. As noted above,when α_(ij) is 0, the constraint equation reduces to a same materialconstraint, as shown on the graph of FIG. 24. At low angles, the vectorV is nearly aligned with the BIDR normal, and thus, all color change isdue to illumination. For values higher than approximately 20°, the valueof α_(ij) is 1. As also noted above, when α_(ij) is 1, the constraintequation reduces to a same illumination constraint (or a smoothillumination, if set by a user in step 1518), as shown on the graph ofFIG. 24. Again, a large angle indicates only a small color differencedue to illumination, since the vector V is not closely aligned with theBIDR normal.

In the distance metric mode, the CPU 12 executes a sigmoid function tocompute α_(ij) as a function of d: α_(ij)=1/1+e^(−s*(d−τ)). A sigmoidslope similar to the slope shown in FIG. 24 will plot the BIDR alignmentweight, α_(ij), against the distance metric d. For low values of d, thedistance between the image locations in the log chromaticity plane issmall, indicating similar material reflectance values for the locations,and thus, corresponding to a same material constraint. Likewise, forhigh values of d, the distance between the image locations in the logchromaticity plane is large, indicating different material reflectancevalues for the locations, and thus, corresponding to a same illuminationconstraint.

In step 1510, the CPU 12 also operates to calculate the confidenceweight, w_(ij), by executing the equation w_(ij)=1−4*α_(ij)*(1−α_(ij)).According to the confidence weight equation, the confidence value is lowin the area of the sigmoid curve at the transition from α_(ij)=0 toα_(ij)=1. Thus, when α_(ij)=0 or α_(ij)=1, the confidence weight w_(ij)is 1, and when α_(ij)=0.5, the confidence weight w_(ij) is 0.

In one exemplary embodiment of the present invention, when there is astrong same material constraint, for example with α_(ij)<0.1, theconstraint is scaled to a higher weight confidence level. The BIDR modelis accurate at predicting a situation when two locations of an image,for example, tokens i and j, are of the same material reflectance.Accordingly, instances when the scalar value α_(ij) is low, indicating asame material constraint, the confidence weight w_(ij) of the constraintis biased by a pre-selected scaling factor, for example, 50.

As illustrated in FIG. 3c , an image pyramid is created in step 1012. Inanother exemplary embodiment of the present invention, the value forw_(ij) is scaled as a function of the pyramid level where the spatialconstraints are written. Inasmuch as higher levels of the pyramid eachhave 1/D² pixels less than the next lower level, constraints at eachhigher level have less impact on the solution. To counteract the 1/D²effect, constraint weights w^(k) _(ij) between image locations, forexample, between tokens i and j, at each level k of the pyramid, arescaled, as follows: w^(k) _(ij)=F_(k)*w_(ij), where the scaling factorF=P*D², P being the processing scale, for example 1. The processingscale specifies the relative strength of constraints written atdifferent levels of the pyramid.

Upon completion of the execution of step 1510 by the CPU 12 to obtainvalues for α_(ij) and w_(ij), the CPU 12 proceeds to step 1526. In step1526, the CPU 12 operates to write the BIDR alignment constraints shownin FIG. 22 with the values computed in steps 1500, 1502 and 1510, forV_(ij), B_(ij), α_(ij) and w_(ij), relative to each of the ab, ac, andbe pairs from the a, b, c tokens of the example of FIG. 22.

Referring once again to FIG. 3c , step 1038 a is executed by the CPU 12by selecting locations from different scales of the scale-spaced imagepyramid shown in FIG. 3b . By writing constraints between pyramid-spacedlocations, the constraints can be applied to a larger range of the imageby writing spatial constraints on higher levels (lower resolution) ofthe pyramid, and also writing downsampling constraints that connect, forexample, tokens at a higher level, to the base level.

Constraints between locations at different levels of the image pyramid,using the color maps created by execution of the routine of FIG. 27, canbe written as anchor constraints. As discussed above, the weightedconstraint, when stated in the context of an anchor constraint, isexpressed as: Mi−Mj=x. In this case, inasmuch as the map created viaexecution of the routine of FIG. 27 results in the same color value foreach of the locations of the different levels forming a constraint, x=0.

In addition, a weight w_(ij) can be assigned to each anchor constraint.Due to the known equality for color values for the two locations of eachanchor constraint, the anchor constraints are strong, and the weightassigned to each constraint is set at a high value to insure that higherlevels of the pyramid have a significant effect on the results at thelower levels.

Referring once again to FIG. 3c , step 1038 d is executed by the CPU 12,in the event the image processing is relative to a video recording, byselecting locations from temporally spaced frames of the video, as shownin FIG. 25b . In a video, each frame can be processed independently fromone another, however, adjacent frames often have a significant amount ofredundancy, and temporal constraints can be used to provide morecoherent and flicker-free videos for corresponding intrinsic materialreflectance and illumination videos. A sliding temporal window is usedrelative to each frame k of the video, and a fixed number of previousframes.

According to a feature of the present invention, temporal constraintscan be implemented according to a color similarity constraint concept,including, in one exemplary embodiment of the present invention, theanchor constraint variation, expressed as follows: w^(n)_(ij)*(Mi−Mj)=w^(n) _(ij)*(α_(ij)V_(ij)+(1−α_(ij))B_(ij)), the samebasic equation as discussed above, however where the tokens i and j arefrom different video frames, token i is from frame k and token j is fromtemporally spaced frame k−n, where n is an integer indicating the numberof frames between k and previous frame k−n, n being greater than orequal to 1, and further, where the temporal constraint weight w^(n)_(ij) is scaled as a function of the distance n between a temporallyspaced frame and frame k.

Referring now to FIG. 29, there is shown a flow chart for a temporalconstraint. In step 1700, the CPU 12 receives as an input token i fromthe image file 18 depicting frame k of a video. The CPU 12 then proceedsto the decision block 1702. In decision block 1702, the CPU 12 checks auser setting as to a mode for temporal constraints. If the mode is setto use BIDR alignment constraints, the CPU 12 proceeds to step 1704, andwrites BIDR alignments constraints. If the mode is not set to use BIDRalignment constraints, the CPU 12 proceeds to step 1706, and writesanchor constraints.

In either mode, the CPU 12 writes the constraints using scaling factorss₁, s₂, . . . , s_(N). The scaling factors determine how much to scalethe confidence weight of a constraint as a function of how many framesthere are between the k frame and the k−n frame of a constraint. Thescaling factors are calculated to insure that frames closer to the kframe have a stronger effect on results in a solve, than frames furtheraway. To that end, each of a temporal sigma σ_(t) (1708) and temporalweight T_(w) (1710) is set by a user and input to the CPU 12. Thetemporal sigma is used to determine how many frames N there are from k−1to k−n. The temporal weight indicates the strength temporal constraintshave relative to spatial constraints.

In step 1712, the CPU 12 uses the temporal sigma to compute N. In anexemplary embodiment of the present invention, the N=round(3*σ_(t)). Instep 1714, the CPU 12 calculates the scaling factors s₁, s₂, . . . ,s_(N), as follows:

s _(n) =T _(w)*exp(−n ²/2σ_(t) ²).

Upon calculation of the scaling factors s₁, s₂, . . . , s_(N), the CPU12 outputs the results to each of steps 1704 and 1706.

When use BIDR alignment constraints is set, the CPU 12 proceeds to step1704. In step 1704, the CPU writes a spatial BIDR alignment constraintfor token i in frame k, and a set of temporal constraints for framesk−1, . . . , k−N, as per the routine of FIG. 23, however with w^(n)_(ij) in each temporal constraint calculated as w^(n)_(ij)=s_(n)*w_(ij).

When use BIDR alignment constraints is not set, the CPU 12 proceeds tostep 1706. In step 1706, the CPU writes anchor constraints between tokeni in frame k and the closest matching token in each of frames k−1, k−2,. . . , k−N, via execution of the routine of FIG. 30. Referring now toFIG. 30, in step 1720, token i is input to the CPU 12. In step 1722, theCPU 12 computes a color flow vector for token i. In an exemplaryembodiment of the present invention, as shown in FIG. 3c , color flowvectors are computed and stored in a color flow vector pyramid (seesteps 1028, 1030 of FIG. 3c , and FIG. 3g ). Accordingly, step 1722 canbe executed via a reference to the color flow pyramid to obtain thecolor flow vector for token i.

In step 1724, the CPU 12 uses the color flow vector to identify acorresponding token j in the k−n frame. In step 1726 the CPU 12 operatesto compute the color difference between token i and token j:|I_(i)−I_(j)|, then proceeds to decision block 1728.

In decision block 1728, the CPU 12 determines if |I_(i)−I_(j)| is lessthan a threshold, for example, a threshold indicating a color differencebetween token i and token j that corresponds to a color similarityweight, α, that is less than 0.1. If yes, the CPU 12 proceeds to step1730.

In step 1730, the CPU 12 writes a strong same material constraintbetween token i and token j: Mi−Mj=0. The CPU 12 then returns theconstraint for concatenation in the [A] [x]=[b] matrix (1732).

If |I_(i)−I_(j)| is greater than the threshold, the CPU 12 then proceedsto step 1734. In step 1734, the CPU 12 writes weakly weighted smoothillumination constraint between token i and token j: Mi−Mj=I_(i)−I_(j).The CPU 12 then returns the constraint for concatenation in the [A][x]=[b] matrix (1732).

In an alternative exemplary embodiment of the present invention, aninternal control can be set to turn off the threshold process. Indecision block 1736, if set to use a hard threshold, the CPU 12 executesfrom decision block 1728. If not set to use a hard threshold, the CPU 12proceeds to step 1738.

In step 1738, the CPU 12 applies a sigmoid function to obtain α as afunction of |I_(i)−I_(j)|, as follows: α=1/1+e^(−s*(|Ii−Ij|−τ)), where sand τ are sigmoid slope and threshold parameters set by a user.

In step 1740, the CPU 12 writes a constraint between token i and tokenj, using the a determined from the sigmoid curve.

In this case, for example, in a weighted constraint, when α<0.1, a samematerial constraint is written, with w_(ij)=1 or higher, and in the casewhen α>0.1, a weak, same illumination constraint is written, with, forexample, w_(ij)=1e⁻⁵. The CPU 12 then returns the constraint forconcatenation in the [A] [x]=[b] matrix (1732).

Moreover, the sigmoid threshold τ can be varied as a function of colorsaturation. A scaling factor is used to vary the sigmoid threshold, withthe scaling factor being set to increase the threshold as colorsaturation increases.

Referring once again to FIG. 3c , step 1038 c is executed by the CPU 12to write constraints based upon user applied scribbles. The CPU 12writes a constraint for each scribble, based upon the type of scribble.For example, in the case of a same reflectance scribble (all pixelsconstituting the scribble are of the same material reflectance), a samematerial constraint is written.

Upon completion of each of steps 1038 a-1038 d, the CPU 12 proceeds tostep 1040. In step 1040, the CPU 12 concatenates all of the constraintsin the [A] [x]=[b] matrix, as shown in the example of FIG. 22. In step1042, the CPU 12 solves the matrix equation, for example, executing theroutine described for the solve of the matrix equation shown in theexample of FIG. 21.

FIG. 31 is a generalized functional block diagram for the serviceprovider 24 constraint builder 26 and solver 30 of FIG. 4. To summarizethe above described constraint examples in a general scheme, a selectionis made of an image 32, (one or more of image file 18, 18 a, . . . 18 n)and, for each selected image 32, a number of constraint generators froma set of constraint generators 1, 2, . . . N, (the constraint generatorsoftware modules) for example, by a user, via the GUI. The set ofconstraint generators 1-N includes the constraints described above, andany additional constraining relationships developed as a function ofspatio-spectral information for an image. The above described set ofconstraints is provided as an example. The present inventioncontemplates any constraining relationship based upon spatio-spectraloperators, that provides a logical deduction regarding material andillumination aspects of an image, and thus a basis for constructingmatrices [A] and [b] to define a set of equations whose optimal solutioncaptures intrinsic illumination and material components of a givenimage.

Likewise, a set of operators 1-M, generated by the Type C tokenizationblock 35 or the operators block 28, includes all operators defined inthe constraint generator modules 1-N. As shown in FIG. 31, the serviceprovider 24 provides all of the operators 1-M, as required by theselected constraint generators 1-N and further couples the selectedconstraint generators 1-N to a constraint assembly 39 via a logicalswitch 40 (both configured within the constraint builder 26). In theevent any of the operators 1-M for a selected image 32 are not alreadystored by the service provider 24, the service provider 24 utilizes theoperators block 28 to compute such operators on demand, in the mannerdescribed above. The constraint assembly 39 constructs a separate [A][x]=[b] matrix for each one of the selected constraint generators, as afunction of the operators and the constraining relationships defined inthe respective constraint generators 1-N. In each case, the [A] [x]=[b]matrix is constructed in a similar manner as described above for thesame illumination and BIDR alignment examples.

Upon completion of the construction of the system of equations [A]_(i)[x]=[b]_(i), for each of the selected constraint generators, i={1, 2, .. . N}, the constraint assembly 39 concatenates the constituent matrices[A]_(i), [b]_(i), from each constraint generator.

Since each of the concatenated equations may contain a different subsetof the unknowns, [x], the assembly is performed such that correspondingcolumns of individual matrices [A]_(i), that constrain particularunknowns in [x], are aligned. The concatenated matrices, [A] [x]=[b],are then input to the solver 30, for solution of the unknowns in thecomplete [x] vector, pursuant to the selected optimization procedure,for output of intrinsic images 34. The individual constraints within theconcatenated matrices, [A] [x]=[b], can be weighted relative to oneanother as a function of factors such as perceived importance of therespective constraint, strength or empirically determined confidencelevel, as, for example, the BIDR alignment and confidence weightsdescribed above.

The above described example of a same illumination constraint utilizesType C token and Type B token spatio-spectral operators. These tokenoperators provide an excellent representation of images that includelarge surface areas of a single material, such as are often depicted inimages including man-made objects. As such, processing of the image at acoarse scale of resolution, for example, at the level of image file 18a, is possible. However, in many natural scenes there are often largeareas of highly textured regions, such as sand, grass, stones, foliage,and so on. Each texture has an intrinsic scale, which can be correlatedto one of the scale representations of image files 18 a, . . . 18 n. Asnoted above, identification of Type B tokens using Type C tokens, can bedifficult in an image texture, particularly at a fine scale ofresolution. According to a further feature of the present invention, atexton histogram operator provides a mechanism for capturingstatistically uniform spatial variations of textured regions in a mannerthat is useful in a constraint based optimization, for example, asexpressed by the [A] [x]=[b] matrix equation.

Thus, according to this feature of the present invention, rather thangenerating Type C tokens in textured regions of an image, from intensityhistograms, for use in identifying Type B tokens, as described above,texture tokens are generated as a species of Type B tokens, for use in aconstraint. Such a texture token analysis can be performed, for example,at a preselected scale suitable to the scale of the texture, for a moreefficient and accurate analysis. In an exemplary embodiment of thetexton histogram operator, the operators block 28 converts each pixel ofthe image (or pixels of those regions of an image identified ascomprising a texture) from the recorded color band representation of therespective image file 18, such as, for example, RGB color band values,to a two band representation wherein the two bands comprise a textonlabel and a texton histogram label. The two band representations for thepixels are then used to identify texture tokens, as will appear.

A texton label for each pixel is generated through execution of aclustering process. A texture can be characterized by a textureprimitive (for example, in a grass texture, a single blade of grass),and the spatial distribution of the primitive. A texton analysis is ananalytical method for characterizing a texture primitive, for examplevia a clustering algorithm. Clustering is a process for locating centersof natural groups or clusters in data. In an exemplary embodiment of thepresent invention, the data comprises pixel patches selected from amongthe pixels of an image being segregated into material and illuminationcomponents. For example, 3×3 pixel patches are clustered into Kdifferent groups, with each group being assigned a designating number(1, 2, 3, . . . K). The texton label for each pixel of the 3×3 array isthe group number of the group to which the respective patch was assignedduring the clustering process.

To expedite execution of a clustering algorithm, random samples of 3×3patches can be selected throughout the image, or region of the imageidentified as comprising a texture, for processing in a clusteringalgorithm. After execution of the clustering algorithm by the CPU 12(operating as the operators block 28), each 3×3 patch of the image isassigned the texton label of the closest one of the K group centersidentified in the clustering process, as executed in respect of theselected random samples.

To advantage, prior to execution of a clustering algorithm, the pixelsof the image are subject to an image intensity normalization. In aclustering process utilizing an intensity-based distance matrix, darkareas of an image may be placed in a single group, resulting in an underrepresentation of groups for shadowed areas of a textured region of animage. A normalization of the image provides a more accurate textonrepresentation for texture regions under varying illumination. Anormalized intensity for a pixel can be expressed by:

i _(norm)(n,m)=log(i(n,m)/i _(b)(n,m)),

where i_(norm)(n,m) is the normalized intensity for a pixel p(n,m),i(n,m) is the intensity for the pixel p(n,m), as recorded in the imagefile 18, and i_(b)(n,m) is a blurred or low passed filtered version ofthe pixel p(n,m). For example, a 10 pixel blur radius can be used in anystandard blurring function.

Clustering can be executed according to any known clustering algorithm,such as, for example, K means clustering where there are K clusters orgroups Si, i=1, 2, . . . K, and i is the mean point or center point ofall the data points x_(j)εS_(i). In our example, each x_(j) comprises aselected 3×3 pixel patch arranged as a 9×1 vector of the nine pixels inthe patch (27 elements total, including the RGB values of each of thenine pixels of the vector). As noted above, each mean point μ_(i) isassigned a texton label, 1, 2, 3 . . . K, that becomes the texton labelfor any pixel of a 3×3 patch clustered into the group for which therespective mean point is the center.

According to an exemplary embodiment of the present invention, the CPU12 executes the algorithm by initially partitioning the selected 9×1vectors, representing 3×3 pixel patches of the image, into K initialgroups Si. The CPU 12 then calculates a center point pi, for each groupSi utilizing an intensity-based distance matrix. After determining acenter point μ_(i), for each group S_(i), the CPU 12 associates each 9×1vector to the closest center point μ_(i), changing groups if necessary.Then the CPU 12 recalculates the center points μ_(i). The CPU 12executes iterations of the steps of associating each 9×1 vector to theclosest center point μ_(i), and recalculating the center points μ_(i),until convergence. Convergence is when there is no need to change thegroup for any of the 9×1 vectors. At that point, the CPU 12 assigns thegroup number for the respective center point μ_(i), as the texton labelfor the pixels of each vector in that group.

As noted above, pixels of 3×3 patches not selected as samples forclustering are assigned the texton label of the closest one of the Kgroup centers pi, identified in the clustering process, as executed inrespect of the selected random samples. A texton label map is stored bythe service provider 24, and is coextensive with the pixel array of FIG.2. In the texton label map, for each pixel location, there is anindication of the respective texton label.

Upon completion of the texton label assignment for pixels of the image,the CPU 12 operates to generate a texton histogram for each pixel toprovide a representation of the spatial variation of textonrepresentations within a textured region of the image. To that end, theCPU 12 accesses the texton label map. At each pixel location within thetexton label map, a pixel patch of, for example, 21×21 pixels, is set uparound the current location. The 21×21 patch size is far greater thanthe 3×3 patch sized used to generate the texton representations, so asto capture the spatial variations of the texture. A texton histogram isthen generated for the pixel location at the center of the 21×21 patch,in a similar manner as the intensity histogram described above. However,rather than bins based upon color band values, in the texton histogram,there is a bin for each texton label value, 1, 2, 3 . . . K. The countfor each bin corresponds to the number of pixels in the 21×21 patchhaving the texton label value for the respective bin.

When a texton histogram is generated for each pixel of the texton labelmap, the CPU 12 executes a second clustering step. In the secondclustering step, the texton histograms are clustered using spectralclustering. Spectral clustering techniques use a spectrum of asimilarity matrix of data of interest, (in our example, the textonhistograms) to reduce the dimensionality for clustering in fewerdimensions. A similarity matrix for a given set of data points A can bedefined as a matrix S where S_(ij) represents a measure of thesimilarity between points i, jεA. In our example, eigenvectors of theLaplacian are clustered using a mean shift. The distance metric is achi-squared distance of the histograms.

A texton histogram label (1, 2 . . . ) is assigned to each cluster groupdefined by the clustering procedure. For each pixel of the texton labelmap, the texton histogram label for the cluster group corresponding to atexton histogram that is nearest the texton histogram for the respectivepixel, is assigned to that pixel. Distance is defined as the chi-squaredhistogram distance. Upon completion of the assignment of a textonhistogram label to each pixel, each pixel is now represented by a twoband, texton label, texton histogram label representation.

According to a feature of the present invention, the two band, textonlabel, texton histogram label representations for pixels of an imagefile 18 (or any of the selectively varied representations 18 a, . . . 18n) can be utilized in a constraint for construction of an [A]_(i)[x]=[b]_(i) constituent within the concatenated matrices, [A] [x]=[b].For example, it can be assumed that a region of an image whereincontiguous pixels within the region all have the same two band, textonlabel, texton histogram label representation, comprises a region of thesame mean material of a texture depicted in the image. Such a region canbe referred to as a texture token, a species of a Type B token. Thus, aconstraint can be imposed that all Type C tokens within the same texturetoken are of the same mean material. In this constraint, the Type Ctokens are the Type C tokens generated from the color band values of theconstituent pixels by the Type C tokenization block 35.

While the above exemplary embodiment of the present invention has beendescribed with a user selecting constraint generators and mathematicaloperations via a GUI, the image segregation processing can be done inother operating modes, such as automatically, with images, constraintgenerators and mathematical operations being automatically selected, forexample, as a function of image parameters.

Referring once again to FIG. 3a , in block 1006, the CPU 12 outputs theintrinsic images generated in the image segregation block 1004. Suchintrinsic images can be advantageously used in such diverse applicationsas object recognition, data compression or for improved results in animage modification. For example, a color correct gamma correction can beachieved by performing an intensity adjustment on the illuminationimage, and merging the intensity adjusted illumination image with thecorresponding material image, for a color correct, intensity adjustedoutput image.

In summary, the selectively varied representation embodiment of thepresent invention provides modes of operation that permitspatio-spectral operators to be computed and applied at any one ofseveral different representations of the image, for example, arepresentation at a scale of resolution that matches the intrinsic scaleof an image feature. Also, different areas of the image can be processedat different scales, such as edge regions at a fine scale, and non-edgeregions at a coarse scale. Moreover, a constraint can be applied over alimited extent of one representation of the image and projected to amore coarse scale to extend the range of the constraint. All of thevarious modes afforded by the selectively varied representationembodiment of the present invention improve the accuracy of an imagesegregation operation, while providing efficiencies that speed up theperformance of the computer system 10. Moreover, the weighted constraintprovides a flexible, weighted constraint arrangement that insuresaccurate results by efficiently accounting for color change due tomaterial reflectance, illumination or a combination of both.

In the preceding specification, the invention has been described withreference to specific exemplary embodiments and examples thereof. Itwill, however, be evident that various modifications and changes may bemade thereto without departing from the broader spirit and scope of theinvention as set forth in the claims that follow. The specification anddrawings are accordingly to be regarded in an illustrative manner ratherthan a restrictive sense.

What is claimed is:
 1. An automated, computerized method for processingan image, comprising the steps of: providing an image file depicting animage having image locations, in a computer memory; selecting imagelocations i and j; setting a constraint between the selected imagelocations i and j according to a relationship defined by abi-illuminant, dichromatic reflection model (BIDR model), the constraintset as a function of a degree of change corresponding to each of anillumination change and/or a material reflectance change according to asigmoid function based upon the BIDR model; identifying an intrinsiccomponent of the selected image locations as a function of theconstraint; and generating an intrinsic image corresponding to the imageas a function of the intrinsic component.
 2. A device which comprises: acomputer; and a memory storing an image file containing an image havingimage locations; the computer is arranged and configured to execute aroutine to select image locations i and j, set a constraint between theselected image locations i and j according to a relationship defined bya bi-illuminant, dichromatic reflection model (BIDR model), theconstraint set as a function of a degree of change corresponding to eachof an illumination change and/or a material reflectance change accordingto a sigmoid function based upon the BIDR model, identify an intrinsiccomponent of the selected image locations as a function of theconstraint, and generate an intrinsic image corresponding to the imageas a function of the intrinsic component.
 3. A computer program product,disposed on a non-transitory computer readable media, the productincluding computer executable process steps operable to control acomputer to: receive an image file depicting an image having imagelocations, select image locations i and j, set a constraint between theselected image locations i and j according to a relationship defined bya bi-illuminant, dichromatic reflection model (BIDR model), theconstraint set as a function of a degree of change corresponding to eachof an illumination change and/or a material reflectance change accordingto a sigmoid function based upon the BIDR model, identify an intrinsiccomponent of the selected image locations as a function of theconstraint, and generate an intrinsic image corresponding to the imageas a function of the intrinsic component.